Nov 03 15:59:50.724084 ip-10-0-137-159 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 15:59:50.724125 ip-10-0-137-159 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 15:59:50.724135 ip-10-0-137-159 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 15:59:50.724627 ip-10-0-137-159 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 16:00:00.911857 ip-10-0-137-159 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 16:00:00.911873 ip-10-0-137-159 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8b91d1e56aa7481087c6fdcf00a77ede --
Nov 03 16:01:52.742227 ip-10-0-137-159 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 16:01:53.175937 ip-10-0-137-159 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:01:53.175937 ip-10-0-137-159 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 16:01:53.175937 ip-10-0-137-159 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:01:53.175937 ip-10-0-137-159 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:01:53.182260 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182169    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:53.182260 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182257    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:53.182260 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182261    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:53.182260 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182265    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:53.182260 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182268    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182271    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182274    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182277    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182281    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182283    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182285    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182288    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182292    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182295    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182297    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182300    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182302    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182305    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182308    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182310    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182313    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182316    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182319    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:53.182446 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182321    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182324    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182327    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182329    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182332    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182336    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182340    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182343    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182346    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182349    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182351    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182354    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182356    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182359    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182362    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182364    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182367    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182369    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182372    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:53.182886 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182374    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182377    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182379    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182382    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182384    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182387    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182389    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182391    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182394    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182396    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182399    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182401    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182404    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182407    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182409    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182412    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182414    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182417    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182419    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:53.183361 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182422    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182425    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182427    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182429    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182432    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182435    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182438    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182441    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182444    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182446    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182449    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182451    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182454    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182457    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182459    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182462    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182464    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182467    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182469    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182472    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:53.183803 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182475    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182477    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182481    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182483    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182486    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182488    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182493    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182496    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182499    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182502    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182505    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182507    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182510    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182513    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182516    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182518    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182521    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182523    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182526    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:53.184281 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182528    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182531    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182534    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182536    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182540    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.182542    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183366    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183374    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183377    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183380    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183383    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183386    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183389    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183391    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183394    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183396    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183399    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183402    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183405    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183407    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:53.184764 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183410    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183413    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183416    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183418    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183421    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183423    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183425    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183428    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183430    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183433    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183435    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183439    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183441    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183444    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183447    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183451    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183456    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183459    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183463    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:53.185266 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183467    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183471    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183473    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183476    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183478    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183481    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183483    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183486    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183488    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183490    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183493    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183496    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183499    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183501    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183504    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183507    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183510    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183512    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183515    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183518    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:53.185717 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183521    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183524    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183527    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183529    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183532    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183534    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183537    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183539    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183542    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183544    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183547    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183549    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183552    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183554    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183556    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183559    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183561    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183564    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183566    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183569    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:53.186215 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183571    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183574    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183577    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183580    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183583    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183585    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183588    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183591    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183593    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183596    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183598    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183601    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183603    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183607    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183609    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183612    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183614    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183617    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183619    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:53.186694 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183622    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183624    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183627    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183630    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183632    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183635    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183637    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183641    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183643    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183646    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183648    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183651    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183653    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.183656    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184175    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184180    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184184    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184187    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184189    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:53.187178 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184192    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184195    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184197    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184200    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184202    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184205    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184207    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184210    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184212    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184215    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184217    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184220    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184223    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184226    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184229    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184231    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184234    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184236    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184239    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184241    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:53.187658 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184244    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184246    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184249    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184252    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184254    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184257    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184260    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184262    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184265    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184267    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184270    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184274    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184278    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184281    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184284    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184287    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184290    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184293    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184296    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:53.188133 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184298    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184301    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184303    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184306    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184308    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184311    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184313    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184315    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184318    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184320    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184323    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184325    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184328    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184331    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184333    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184336    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184338    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184341    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184343    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184346    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:53.188559 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184349    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184351    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184354    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184357    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184360    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184362    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184365    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184367    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184370    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184372    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184374    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184377    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184379    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184381    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184384    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184386    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184389    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184391    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184393    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184396    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:53.189080 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184398    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184401    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184403    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184405    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184408    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184411    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184413    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184416    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184418    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184421    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184423    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184426    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184428    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184431    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184435    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184438    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184441    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184445    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184447    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:53.189592 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184449    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184452    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.184454    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185433    2577 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185447    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185456    2577 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185461    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185465    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185468    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185472    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185480    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185483    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185486    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185490    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185493    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185496    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185500    2577 flags.go:64] FLAG: --cgroup-root=""
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185502    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185505    2577 flags.go:64] FLAG: --client-ca-file=""
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185508    2577 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185511    2577 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185517    2577 flags.go:64] FLAG: --cluster-domain=""
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185520    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185523    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 16:01:53.190044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185526    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185529    2577 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185536    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185539    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185542    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185546    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185549    2577 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185552    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185555    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185559    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185561    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185565    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185568    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185571    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185574    2577 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185577    2577 flags.go:64] FLAG: --enable-server="true"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185580    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185584    2577 flags.go:64] FLAG: --event-burst="100"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185588    2577 flags.go:64] FLAG: --event-qps="50"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185590    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185593    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185596    2577 flags.go:64] FLAG: --eviction-hard=""
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185600    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185603    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185606    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 16:01:53.190625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185609    2577 flags.go:64] FLAG: --eviction-soft=""
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185612    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185615    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185618    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185620    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185623    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185626    2577 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185629    2577 flags.go:64] FLAG: --feature-gates=""
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185633    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185637    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185641    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185644    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185647    2577 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185650    2577 flags.go:64] FLAG: --help="false"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185653    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185656    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185659    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185662    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185665    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185669    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185672    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185674    2577 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185677    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 16:01:53.191238 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185680    2577 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185683    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185686    2577 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185688    2577 flags.go:64] FLAG: --kube-reserved=""
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185691    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185694    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185697    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185699    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185702    2577 flags.go:64] FLAG: --lock-file=""
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185705    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185708    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185710    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185716    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185719    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185722    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185725    2577 flags.go:64] FLAG: --logging-format="text"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185728    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185731    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185735    2577 flags.go:64] FLAG: --manifest-url=""
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185738    2577 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185743    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185746    2577 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185749    2577 flags.go:64] FLAG: --max-pods="110"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185752    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185755    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 16:01:53.192016 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185758    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185761    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185764    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185767    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185770    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185777    2577 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185780    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185783    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185786    2577 flags.go:64] FLAG: --pod-cidr=""
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185789    2577 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185791    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185794    2577 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185798    2577 flags.go:64] FLAG: --port="10250"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185801    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185803    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0271e572ab51e3f93"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185806    2577 flags.go:64] FLAG: --qos-reserved=""
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185809    2577 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185812    2577 flags.go:64] FLAG: --register-node="true"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185815    2577 flags.go:64] FLAG: --register-with-taints=""
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185818    2577 flags.go:64] FLAG: --registry-burst="10"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185821    2577 flags.go:64] FLAG: --registry-qps="5"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185824    2577 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185826    2577 flags.go:64] FLAG: --reserved-memory=""
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185830    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185833    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185837    2577 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 16:01:53.193218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185840    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185843    2577 flags.go:64] FLAG: --runonce="false"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185846    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185849    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185851    2577 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185854    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185857    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185861    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185864    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185866    2577 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185869    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185872    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185874    2577 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185877    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185880    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185883    2577 flags.go:64] FLAG: --system-cgroups=""
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185885    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185890    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185897    2577 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185899    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185906    2577 flags.go:64] FLAG: --tls-min-version=""
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185909    2577 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185912    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185915    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185917    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 16:01:53.194302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185920    2577 flags.go:64] FLAG: --v="2"
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185924    2577 flags.go:64] FLAG: --version="false"
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185928    2577 flags.go:64] FLAG: --vmodule=""
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185932    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.185935    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186106    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186112    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186115    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186118    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186121    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186123    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186126    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186129    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186131    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186134    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186137    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186140    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186142    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186145    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186147    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186150    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:53.195471 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186152    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186155    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186157    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186159    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186163    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186166    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186168    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186171    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186173    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186176    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186179    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186181    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186184    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186186    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186189    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186191    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186193    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186197    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186200    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:53.196357 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186202    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186205    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186207    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186210    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186213    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186215    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186218    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186220    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186224    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186226    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186229    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186231    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186234    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186237    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186239    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186242    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186245    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186250    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186253    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:53.196959 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186256    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186259    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186261    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186264    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186266    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186268    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186271    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186274    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186276    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186279    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186281    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186285    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186288    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186290    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186294    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186297    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186300    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186303    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186306    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:53.197747 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186309    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186311    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186314    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186317    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186319    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186322    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186324    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186327    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186329    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186332    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186335    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186339    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186342    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186344    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186347    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186349    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186351    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186354    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186356    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186359    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:53.198404 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186361    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186364    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186367    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186369    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186373    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186375    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186378    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186381    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186383    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186385    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186388    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186391    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186393    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.186420    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186465    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186471    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:53.199091 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186494    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186497    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186500    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186503    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186506    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186508    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186511    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186513    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186518    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186520    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186523    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186526    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186528    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186531    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186533    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186535    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186538    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186540    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186543    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186545    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:53.199517 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186549    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186551    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186554    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186556    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186559    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186561    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186564    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186566    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186569    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186571    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186574    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186577    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186579    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186582    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186585    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186587    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186590    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186592    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186595    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186597    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:53.200067 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186601    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186604    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186606    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186609    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186612    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186614    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186617    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186619    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186621    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186624    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186626    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186629    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186633    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186636    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186638    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186641    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186643    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186645    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186648    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:53.200527 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186650    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186653    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186655    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186658    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186661    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186663    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186665    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186669    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186671    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186673    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186676    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186679    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186681    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186685    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186687    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186690    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186693    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186695    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186698    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:53.201047 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186701    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186703    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186706    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186708    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186711    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186713    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186717    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186719    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186722    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186724    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186729    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186732    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186735    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186738    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186741    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186744    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186746    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186748    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186751    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:53.201498 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186753    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186756    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186759    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186761    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186764    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186766    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.186768    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.194079    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.194095    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.194117    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.194124    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194213    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194221    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194227    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194232    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194238    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194242    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194246    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194251    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194255    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:53.201981 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194259    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194264    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194268    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194274    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194278    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194283    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194287    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194291    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194295    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194299    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194303    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194306    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194311    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194315    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194319    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194323    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194327    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194332    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194337    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194342    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:53.202477 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194348    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194353    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194357    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194362    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194366    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194370    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194374    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194378    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194382    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194386    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194391    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194396    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194400    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194404    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194410    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194414    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194441    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194447    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194453    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194458    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:53.203100 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194462    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194466    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194470    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194475    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194480    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194484    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194488    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194494    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194498    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194502    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194507    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194512    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194516    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194521    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194525    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194530    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194534    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194538    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194542    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:53.203902 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194546    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194551    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194555    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194559    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194563    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194568    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194572    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194576    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194582    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194588    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194593    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194600    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194607    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194611    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194616    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194620    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194625    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194629    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194633    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:53.204699 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194637    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194641    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194646    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194651    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194655    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194659    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194663    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194667    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194672    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194676    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194680    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194684    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194689    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194693    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194697    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194701    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194706    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194710    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:53.205314 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.194714    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.194757    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195010    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195021    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195026    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195030    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195034    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195039    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195043    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195047    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195051    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195056    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195060    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195065    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195071    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195076    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:53.205755 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195081    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195087    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195091    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195095    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195100    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195104    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195108    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195113    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195117    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195121    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195125    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195129    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195134    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195138    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195142    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195146    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195150    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195154    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195159    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:53.206259 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195164    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195168    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195172    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195176    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195181    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195185    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195189    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195193    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195197    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195202    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195206    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195210    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195215    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195219    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195223    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195228    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195233    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195237    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195241    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195246    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:53.206875 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195250    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195254    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195259    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195263    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195267    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195271    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195275    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195279    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195290    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195295    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195300    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195305    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195309    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195313    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195317    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195321    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195325    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195329    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195333    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195337    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:53.207377 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195341    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195346    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195351    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195358    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195362    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195366    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195370    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195374    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195379    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195386    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195390    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195395    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195399    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195403    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195408    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195412    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195416    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195420    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195424    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:53.207855 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195428    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:53.208384 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195433    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:53.208384 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195437    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:53.208384 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195441    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:53.208384 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195446    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:53.208384 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195450    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:53.208384 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195454    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:53.208384 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195461    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:53.208384 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195465    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:53.208384 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195470    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:53.208384 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195474    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:53.208384 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195478    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:53.208384 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195482    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:53.208384 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195486    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:53.208384 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.195531    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:01:53.208384 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195587    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195594    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195598    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195606    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195613    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195618    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195623    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195631    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195636    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195640    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195645    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195649    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195653    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195657    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195661    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195665    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195670    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195674    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195678    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:53.208741 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195683    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195687    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195694    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195700    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195706    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195710    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195714    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195718    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195722    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195727    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195731    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195735    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195740    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195744    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195748    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195752    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195756    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195760    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195765    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:53.209208 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195769    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195774    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195779    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195783    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195787    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195792    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195796    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195801    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195805    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195811    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195815    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195819    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195824    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195828    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195832    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195837    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195841    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195845    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195849    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195854    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:53.209643 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195858    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195863    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195867    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195872    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195876    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195880    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195884    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195888    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195892    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195896    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195900    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195905    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195909    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195915    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195919    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195925    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195929    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195933    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195957    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195962    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:53.210158 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195966    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195971    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195976    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195981    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195985    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195989    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195993    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.195998    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196002    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196006    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196011    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196015    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196019    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196023    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196027    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196032    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196036    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196041    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196045    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:53.210615 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196049    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:53.211071 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196053    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:53.211071 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196057    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:53.211071 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196062    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:53.211071 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196066    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:53.211071 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196070    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:53.211071 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196079    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:53.211071 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196084    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:53.211071 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:53.196088    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:53.211071 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.196163    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 16:01:53.211071 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.198801    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 16:01:53.211071 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.199867    2577 server.go:1014] "Starting client certificate rotation"
Nov 03 16:01:53.211071 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.199975    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 16:01:53.211071 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.200017    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 16:01:53.211071 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.203606    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 16:01:53.211071 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.208051    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 16:01:53.220039 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.220021    2577 log.go:25] "Validated CRI v1 runtime API"
Nov 03 16:01:53.225126 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.225107    2577 log.go:25] "Validated CRI v1 image API"
Nov 03 16:01:53.225900 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.225876    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 16:01:53.228311 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.228286    2577 fs.go:135] Filesystem UUIDs: map[4d2cae4f-9a11-440d-95af-4c8739cc81ee:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c0570534-d3c5-47dd-9c86-ebbf3feea646:/dev/nvme0n1p3]
Nov 03 16:01:53.228374 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.228306    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 16:01:53.233389 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.233281    2577 manager.go:217] Machine: {Timestamp:2026-11-03 16:01:53.231914373 +0000 UTC m=+0.359996921 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3113031 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2358ba4325e67b13944d012380d40a SystemUUID:ec2358ba-4325-e67b-1394-4d012380d40a BootID:8b91d1e5-6aa7-4810-87c6-fdcf00a77ede Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:da:17:71:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:da:17:71:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:fb:16:1d:65:5f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 16:01:53.233389 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.233388    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 16:01:53.233495 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.233461    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 16:01:53.236406 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.236378    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 16:01:53.236535 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.236410    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-159.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 16:01:53.237579 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.237569    2577 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 16:01:53.237618 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.237582    2577 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 16:01:53.237618 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.237594    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 16:01:53.237618 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.237604    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 16:01:53.238353 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.238343    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 16:01:53.241780 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.241767    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 16:01:53.241902 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.241893    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 16:01:53.241929 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.241923    2577 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 16:01:53.241976 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.241932    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 16:01:53.241976 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.241961    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 16:01:53.242025 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.241985    2577 kubelet.go:405] "Adding apiserver pod source"
Nov 03 16:01:53.242025 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.241996    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 16:01:53.242357 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.242346    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 16:01:53.242656 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.242634    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 16:01:53.244809 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.244793    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 16:01:53.250530 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.249922    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 16:01:53.251065 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.251050    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 16:01:53.251140 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.251092    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 16:01:53.251298 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.251277    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 16:01:53.251357 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.251305    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 16:01:53.251357 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.251315    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 16:01:53.251357 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.251324    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 16:01:53.251357 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.251346    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 16:01:53.251357 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.251357    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 16:01:53.251991 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.251981    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 16:01:53.252040 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.251995    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 16:01:53.252040 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.252007    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 16:01:53.252040 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.252018    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 16:01:53.252040 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.252034    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 16:01:53.252197 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.252049    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 16:01:53.253223 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.253205    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 16:01:53.253275 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.253238    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 16:01:53.257079 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.257065    2577 server.go:1263] "Started kubelet"
Nov 03 16:01:53.257159 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.257131    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 16:01:53.257224 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.257150    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 16:01:53.257224 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.257218    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 16:01:53.257899 ip-10-0-137-159 systemd[1]: Started Kubernetes Kubelet.
Nov 03 16:01:53.258317 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.258298    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 16:01:53.259490 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.259468    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-159.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 16:01:53.259490 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.259484    2577 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 16:01:53.262494 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.262476    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 16:01:53.262897 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.262883    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jszch"
Nov 03 16:01:53.263202 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.263183    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 16:01:53.263753 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.263739    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 16:01:53.263882 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.263842    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-159.us-east-2.compute.internal\" not found"
Nov 03 16:01:53.264335 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.264316    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 16:01:53.264335 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.264317    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 16:01:53.264478 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.264342    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 16:01:53.264478 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.264427    2577 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 16:01:53.264478 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.264434    2577 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 16:01:53.264626 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.264614    2577 factory.go:55] Registering systemd factory
Nov 03 16:01:53.264654 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.264629    2577 factory.go:223] Registration of the systemd container factory successfully
Nov 03 16:01:53.264828 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.264818    2577 factory.go:153] Registering CRI-O factory
Nov 03 16:01:53.264828 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.264829    2577 factory.go:223] Registration of the crio container factory successfully
Nov 03 16:01:53.264895 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.264867    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 16:01:53.264895 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.264881    2577 factory.go:103] Registering Raw factory
Nov 03 16:01:53.264895 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.264890    2577 manager.go:1199] Started watching for new ooms in manager
Nov 03 16:01:53.265866 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.265850    2577 manager.go:319] Starting recovery of all containers
Nov 03 16:01:53.274110 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.274082    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-159.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 16:01:53.274237 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.274214    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-159.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 16:01:53.274368 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.274344    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 16:01:53.275596 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.275563    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jszch"
Nov 03 16:01:53.276100 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.273880    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-159.us-east-2.compute.internal.18e4946508974a71  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-159.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-159.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 16:01:53.257040497 +0000 UTC m=+0.385123045,LastTimestamp:2026-11-03 16:01:53.257040497 +0000 UTC m=+0.385123045,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-159.us-east-2.compute.internal,}"
Nov 03 16:01:53.279302 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.279103    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 16:01:53.280138 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.280120    2577 manager.go:324] Recovery completed
Nov 03 16:01:53.284830 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.284817    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:01:53.286536 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.286521    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:01:53.286636 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.286554    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:01:53.286636 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.286569    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:01:53.287094 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.287082    2577 cpu_manager.go:226] "Starting" policy="none"
Nov 03 16:01:53.287148 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.287095    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 16:01:53.287148 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.287085    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:01:53.287148 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.287114    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 16:01:53.287786 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.287770    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:01:53.287858 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.287801    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:01:53.287858 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.287815    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:01:53.289177 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.289164    2577 policy_none.go:50] "Start"
Nov 03 16:01:53.289177 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.289179    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 16:01:53.289276 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.289189    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 16:01:53.291189 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.291179    2577 policy_none.go:44] "Start"
Nov 03 16:01:53.323967 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.323923    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 16:01:53.326888 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.326876    2577 manager.go:341] "Starting Device Plugin manager"
Nov 03 16:01:53.327383 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.327369    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 16:01:53.327420 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.327384    2577 server.go:87] "Starting device plugin registration server"
Nov 03 16:01:53.327587 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.327578    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 16:01:53.327622 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.327589    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 16:01:53.327720 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.327705    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 16:01:53.327832 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.327793    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 16:01:53.327832 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.327803    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 16:01:53.328347 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.328326    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 16:01:53.328412 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.328368    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-159.us-east-2.compute.internal\" not found"
Nov 03 16:01:53.428409 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.428356    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:01:53.430337 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.429245    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:01:53.430337 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.429276    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:01:53.430337 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.429294    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:01:53.430337 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.429323    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.437744 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.436634    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.437744 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.436790    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-159.us-east-2.compute.internal\": node \"ip-10-0-137-159.us-east-2.compute.internal\" not found"
Nov 03 16:01:53.471214 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.471194    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-159.us-east-2.compute.internal\" not found"
Nov 03 16:01:53.481502 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.481476    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 16:01:53.482819 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.482802    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 16:01:53.482897 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.482823    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 16:01:53.482897 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.482841    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 16:01:53.482897 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.482880    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 16:01:53.490546 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.490523    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:01:53.571441 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.571416    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-159.us-east-2.compute.internal\" not found"
Nov 03 16:01:53.583614 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.583590    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal"]
Nov 03 16:01:53.583671 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.583651    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:01:53.584533 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.584517    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:01:53.584593 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.584548    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:01:53.584593 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.584558    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:01:53.586773 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.586761    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:01:53.586950 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.586923    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.587020 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.586974    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:01:53.587519 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.587504    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:01:53.587589 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.587535    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:01:53.587589 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.587550    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:01:53.587654 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.587551    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:01:53.587684 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.587656    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:01:53.587684 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.587671    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:01:53.589521 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.589504    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.589606 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.589534    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:01:53.590256 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.590241    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:01:53.590322 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.590273    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:01:53.590322 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.590289    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:01:53.621317 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.621302    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-159.us-east-2.compute.internal\" not found" node="ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.623297 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.623275    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-159.us-east-2.compute.internal\" not found" node="ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.665928 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.665908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f898674fe0c94d2ff8a087a4cd76c349-config\") pod \"kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"f898674fe0c94d2ff8a087a4cd76c349\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.665999 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.665932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56a59496f84ad599eee942e332b42fd6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"56a59496f84ad599eee942e332b42fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.665999 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.665964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56a59496f84ad599eee942e332b42fd6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"56a59496f84ad599eee942e332b42fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.672356 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.672341    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-159.us-east-2.compute.internal\" not found"
Nov 03 16:01:53.766506 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.766447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f898674fe0c94d2ff8a087a4cd76c349-config\") pod \"kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"f898674fe0c94d2ff8a087a4cd76c349\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.766506 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.766475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56a59496f84ad599eee942e332b42fd6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"56a59496f84ad599eee942e332b42fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.766506 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.766494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56a59496f84ad599eee942e332b42fd6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"56a59496f84ad599eee942e332b42fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.766662 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.766540    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f898674fe0c94d2ff8a087a4cd76c349-config\") pod \"kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"f898674fe0c94d2ff8a087a4cd76c349\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.766662 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.766562    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56a59496f84ad599eee942e332b42fd6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"56a59496f84ad599eee942e332b42fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.766662 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.766540    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56a59496f84ad599eee942e332b42fd6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"56a59496f84ad599eee942e332b42fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.772550 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.772532    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-159.us-east-2.compute.internal\" not found"
Nov 03 16:01:53.873013 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.872993    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-159.us-east-2.compute.internal\" not found"
Nov 03 16:01:53.925201 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.925169    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.926273 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:53.926259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:53.973717 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:53.973693    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-159.us-east-2.compute.internal\" not found"
Nov 03 16:01:54.074228 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:54.074172    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-159.us-east-2.compute.internal\" not found"
Nov 03 16:01:54.121880 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:54.121852    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:01:54.175255 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:54.175231    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-159.us-east-2.compute.internal\" not found"
Nov 03 16:01:54.199350 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:54.199332    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 16:01:54.199758 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:54.199456    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 16:01:54.199758 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:54.199505    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 16:01:54.274848 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:54.274826    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:01:54.281089 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:54.281048    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 15:56:54 +0000 UTC" deadline="2028-05-18 08:07:43.5327382 +0000 UTC"
Nov 03 16:01:54.281089 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:54.281084    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13480h5m49.251656899s"
Nov 03 16:01:54.302260 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:54.302239    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:01:54.364546 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:54.364494    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:54.375836 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:54.375818    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:01:54.376822 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:54.376809    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:01:54.388098 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:54.388082    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:01:54.524569 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:54.524537    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a59496f84ad599eee942e332b42fd6.slice/crio-a1fe67ccc0ffbc19e2e96cbc485098416398466cf149c1884a554c830799fa00 WatchSource:0}: Error finding container a1fe67ccc0ffbc19e2e96cbc485098416398466cf149c1884a554c830799fa00: Status 404 returned error can't find the container with id a1fe67ccc0ffbc19e2e96cbc485098416398466cf149c1884a554c830799fa00
Nov 03 16:01:54.530205 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:54.530184    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:01:54.534777 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:54.534753    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf898674fe0c94d2ff8a087a4cd76c349.slice/crio-d0fb0377af2f674020219c26fa4a1f42c72cea83140ef2909ceca1f643b17f94 WatchSource:0}: Error finding container d0fb0377af2f674020219c26fa4a1f42c72cea83140ef2909ceca1f643b17f94: Status 404 returned error can't find the container with id d0fb0377af2f674020219c26fa4a1f42c72cea83140ef2909ceca1f643b17f94
Nov 03 16:01:56.544953 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.544921    2577 apiserver.go:52] "Watching apiserver"
Nov 03 16:01:56.552672 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.552609    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 16:01:56.553008 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.552982    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-7f6nb","kube-system/global-pull-secret-syncer-qmd98","kube-system/konnectivity-agent-hdgx7","kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-vsg95","openshift-multus/multus-n4g6g","openshift-network-operator/iptables-alerter-5flz8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d","openshift-image-registry/node-ca-j5r6g","openshift-multus/network-metrics-daemon-sf4cc","openshift-network-diagnostics/network-check-target-2ht7q","openshift-ovn-kubernetes/ovnkube-node-2vtl4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-rlgrt"]
Nov 03 16:01:56.555444 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.555420    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:01:56.555552 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:56.555507    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:01:56.557625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.557519    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:01:56.557625 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:56.557596    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:01:56.559984 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.559963    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:01:56.562660 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.562640    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.563961 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.563941    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c56qh\""
Nov 03 16:01:56.564059 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.564045    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 16:01:56.564119 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.564060    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 16:01:56.565280 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.564840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.565783 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.565686    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 16:01:56.567777 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.567353    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.569615 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.569597    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:01:56.572451 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.571854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.574612 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.574592    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:01:56.574700 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:56.574662    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:01:56.576381 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-tuned\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.576477 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-netns\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.576477 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13d056b1-85f8-48eb-b9bf-aeb4b98a8170-agent-certs\") pod \"konnectivity-agent-hdgx7\" (UID: \"13d056b1-85f8-48eb-b9bf-aeb4b98a8170\") " pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:01:56.576477 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cnibin\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.576632 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qrc5\" (UniqueName: \"kubernetes.io/projected/c0652c5d-5440-4b54-9220-fbf77c06ec2f-kube-api-access-8qrc5\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.576632 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-cni-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.576632 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-cni-bin\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.576632 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysctl-d\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.576632 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-systemd\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.576632 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-k8s-cni-cncf-io\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-host\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576692    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c44c5cc-7fb0-40c7-970b-add5c87c0770-tmp\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576717    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwxn6\" (UniqueName: \"kubernetes.io/projected/1c44c5cc-7fb0-40c7-970b-add5c87c0770-kube-api-access-wwxn6\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576763    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-cni-multus\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-conf-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-etc-kubernetes\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13d056b1-85f8-48eb-b9bf-aeb4b98a8170-konnectivity-ca\") pod \"konnectivity-agent-hdgx7\" (UID: \"13d056b1-85f8-48eb-b9bf-aeb4b98a8170\") " pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-cnibin\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-kubelet\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.576994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-daemon-config\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-kubernetes\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysctl-conf\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.577076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-sys\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dbf1e42-f480-47a4-8382-6207e2fe7774-cni-binary-copy\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-hostroot\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hsrw\" (UniqueName: \"kubernetes.io/projected/a00bade6-34ba-4d87-a740-9727b8372650-kube-api-access-9hsrw\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-modprobe-d\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysconfig\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-var-lib-kubelet\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-socket-dir-parent\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-multus-certs\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn24w\" (UniqueName: \"kubernetes.io/projected/1dbf1e42-f480-47a4-8382-6207e2fe7774-kube-api-access-xn24w\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4774\" (UniqueName: \"kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774\") pod \"network-check-target-2ht7q\" (UID: \"af63014e-5ed6-4213-8838-529749c931cb\") " pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-run\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-system-cni-dir\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cni-binary-copy\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-system-cni-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:01:56.578191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.577506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-lib-modules\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.579501 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.579482    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:01:56.579905 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.579883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:01:56.581226 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.581207    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.623252 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.623227    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 16:01:56.642361 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.642341    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-btc8l"
Nov 03 16:01:56.660101 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.660078    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-btc8l"
Nov 03 16:01:56.664606 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.664506    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4r589\""
Nov 03 16:01:56.664606 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.664521    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 16:01:56.664824 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.664654    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 16:01:56.664910 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.664893    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 16:01:56.664972 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.664914    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 16:01:56.664972 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.664893    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 16:01:56.664972 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.664914    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 16:01:56.665125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665060    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5rhh2\""
Nov 03 16:01:56.665189 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665129    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:01:56.665189 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665177    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2h492\""
Nov 03 16:01:56.665287 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665275    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 16:01:56.665329 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665291    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 16:01:56.665329 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665292    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 16:01:56.665329 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665319    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lgwbg\""
Nov 03 16:01:56.665329 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665326    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 16:01:56.665471 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665351    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 16:01:56.665471 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665367    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 16:01:56.665471 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665373    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 16:01:56.665471 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665367    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:01:56.665471 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665378    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 16:01:56.665471 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665398    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qk9pq\""
Nov 03 16:01:56.665471 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665410    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-drdlv\""
Nov 03 16:01:56.665471 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665422    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 16:01:56.665471 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665439    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 16:01:56.665882 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665503    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 16:01:56.665882 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665508    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 16:01:56.665882 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665517    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 16:01:56.665882 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665546    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 16:01:56.665882 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665553    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 16:01:56.665882 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.665559    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 16:01:56.666443 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.666419    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dchj7\""
Nov 03 16:01:56.666731 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.666712    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jdvsk\""
Nov 03 16:01:56.666856 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.666829    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 16:01:56.667189 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.667174    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 16:01:56.678119 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-systemd\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.678216 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.678216 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7ea5065d-e1c6-4152-842c-833cd8155772-dbus\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:01:56.678319 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678240    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-systemd\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.678319 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-systemd\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.678319 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-k8s-cni-cncf-io\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.678422 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-host\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.678422 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.678422 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.678422 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678396    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-k8s-cni-cncf-io\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.678422 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-socket-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.678615 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678452    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-ovnkube-config\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.678615 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678459    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-host\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.678615 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-conf-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.678615 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-etc-kubernetes\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.678615 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13d056b1-85f8-48eb-b9bf-aeb4b98a8170-konnectivity-ca\") pod \"konnectivity-agent-hdgx7\" (UID: \"13d056b1-85f8-48eb-b9bf-aeb4b98a8170\") " pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:01:56.678615 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-cnibin\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.678615 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-daemon-config\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.678615 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678608    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-conf-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678650    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-cnibin\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678676    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-etc-kubernetes\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysctl-conf\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9jz2\" (UniqueName: \"kubernetes.io/projected/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-kube-api-access-x9jz2\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-log-socket\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-modprobe-d\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-var-lib-kubelet\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cni-binary-copy\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-socket-dir-parent\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn24w\" (UniqueName: \"kubernetes.io/projected/1dbf1e42-f480-47a4-8382-6207e2fe7774-kube-api-access-xn24w\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4774\" (UniqueName: \"kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774\") pod \"network-check-target-2ht7q\" (UID: \"af63014e-5ed6-4213-8838-529749c931cb\") " pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-run\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7ea5065d-e1c6-4152-842c-833cd8155772-kubelet-config\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-system-cni-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-node-log\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.678993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13d056b1-85f8-48eb-b9bf-aeb4b98a8170-agent-certs\") pod \"konnectivity-agent-hdgx7\" (UID: \"13d056b1-85f8-48eb-b9bf-aeb4b98a8170\") " pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.678908    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysctl-conf\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679051    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-var-lib-kubelet\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkcbp\" (UniqueName: \"kubernetes.io/projected/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-kube-api-access-qkcbp\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679130    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-run\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679142    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-modprobe-d\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679172    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-socket-dir-parent\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-run-netns\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679208    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-system-cni-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:56.679238    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-cni-bin\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:56.679305    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs podName:a00bade6-34ba-4d87-a740-9727b8372650 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:57.179285281 +0000 UTC m=+3.005514712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs") pod "network-metrics-daemon-sf4cc" (UID: "a00bade6-34ba-4d87-a740-9727b8372650") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679351    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-env-overrides\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679399    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-cni-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-cni-bin\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysctl-d\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679524    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-cni-bin\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.679716 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679547    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-cni-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-registration-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-sys-fs\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679658    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysctl-d\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679721    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-daemon-config\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13ece33e-8a7b-418f-aa9e-eaa2fed48273-hosts-file\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679729    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13d056b1-85f8-48eb-b9bf-aeb4b98a8170-konnectivity-ca\") pod \"konnectivity-agent-hdgx7\" (UID: \"13d056b1-85f8-48eb-b9bf-aeb4b98a8170\") " pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679769    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ece33e-8a7b-418f-aa9e-eaa2fed48273-tmp-dir\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c44c5cc-7fb0-40c7-970b-add5c87c0770-tmp\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwxn6\" (UniqueName: \"kubernetes.io/projected/1c44c5cc-7fb0-40c7-970b-add5c87c0770-kube-api-access-wwxn6\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-etc-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-cni-multus\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679951    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-cni-multus\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679900    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.679961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.680472 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-host-slash\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-cni-netd\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-kubelet\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680157    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680181    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-kubelet\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-kubernetes\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-sys\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680227    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-kubernetes\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-iptables-alerter-script\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-serviceca\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680271    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-sys\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/596632c1-fb08-4008-ad12-7d399fa3994b-ovn-node-metrics-cert\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dbf1e42-f480-47a4-8382-6207e2fe7774-cni-binary-copy\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-hostroot\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hsrw\" (UniqueName: \"kubernetes.io/projected/a00bade6-34ba-4d87-a740-9727b8372650-kube-api-access-9hsrw\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysconfig\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680450    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-hostroot\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.681241 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-system-cni-dir\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qrc5\" (UniqueName: \"kubernetes.io/projected/c0652c5d-5440-4b54-9220-fbf77c06ec2f-kube-api-access-8qrc5\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-systemd-units\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680534    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysconfig\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-multus-certs\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680564    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-system-cni-dir\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-device-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-kubelet\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-lib-modules\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680637    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-multus-certs\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-tuned\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs5sr\" (UniqueName: \"kubernetes.io/projected/13ece33e-8a7b-418f-aa9e-eaa2fed48273-kube-api-access-rs5sr\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680755    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-lib-modules\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680832    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7s62\" (UniqueName: \"kubernetes.io/projected/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-kube-api-access-v7s62\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680846    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dbf1e42-f480-47a4-8382-6207e2fe7774-cni-binary-copy\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-slash\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680935    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.682200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-ovn\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.683004 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-ovnkube-script-lib\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.683004 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.681024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-netns\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.683004 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.681062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cnibin\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.683004 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.681123    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-netns\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.683004 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.681160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:01:56.683004 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.681187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-host\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:01:56.683004 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.680774    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cni-binary-copy\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.683004 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.681214    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cnibin\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.683004 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.681281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-var-lib-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.683004 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.681313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.683004 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.681342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svwtn\" (UniqueName: \"kubernetes.io/projected/596632c1-fb08-4008-ad12-7d399fa3994b-kube-api-access-svwtn\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.683004 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.681524    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.684015 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.683987    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-tuned\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.684111 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.684042    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c44c5cc-7fb0-40c7-970b-add5c87c0770-tmp\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.684746 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.684714    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13d056b1-85f8-48eb-b9bf-aeb4b98a8170-agent-certs\") pod \"konnectivity-agent-hdgx7\" (UID: \"13d056b1-85f8-48eb-b9bf-aeb4b98a8170\") " pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:01:56.691164 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:56.691144    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:01:56.691426 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:56.691167    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:01:56.691426 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:56.691179    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r4774 for pod openshift-network-diagnostics/network-check-target-2ht7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:56.691426 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:56.691232    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774 podName:af63014e-5ed6-4213-8838-529749c931cb nodeName:}" failed. No retries permitted until 2026-11-03 16:01:57.191216249 +0000 UTC m=+3.017445668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r4774" (UniqueName: "kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774") pod "network-check-target-2ht7q" (UID: "af63014e-5ed6-4213-8838-529749c931cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:56.693375 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.693348    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn24w\" (UniqueName: \"kubernetes.io/projected/1dbf1e42-f480-47a4-8382-6207e2fe7774-kube-api-access-xn24w\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.694410 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.694364    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qrc5\" (UniqueName: \"kubernetes.io/projected/c0652c5d-5440-4b54-9220-fbf77c06ec2f-kube-api-access-8qrc5\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.695379 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.695039    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hsrw\" (UniqueName: \"kubernetes.io/projected/a00bade6-34ba-4d87-a740-9727b8372650-kube-api-access-9hsrw\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:01:56.695379 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.695350    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwxn6\" (UniqueName: \"kubernetes.io/projected/1c44c5cc-7fb0-40c7-970b-add5c87c0770-kube-api-access-wwxn6\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.781938 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.781906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-systemd-units\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.782112 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.781945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-device-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.782112 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.781969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-kubelet\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.782112 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.781992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs5sr\" (UniqueName: \"kubernetes.io/projected/13ece33e-8a7b-418f-aa9e-eaa2fed48273-kube-api-access-rs5sr\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:01:56.782112 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7s62\" (UniqueName: \"kubernetes.io/projected/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-kube-api-access-v7s62\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:01:56.782112 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-slash\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.782112 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.782112 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-ovn\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.782112 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-ovnkube-script-lib\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.782481 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:01:56.782481 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-host\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:01:56.782481 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-var-lib-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.782481 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.782481 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svwtn\" (UniqueName: \"kubernetes.io/projected/596632c1-fb08-4008-ad12-7d399fa3994b-kube-api-access-svwtn\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.782481 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.782481 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7ea5065d-e1c6-4152-842c-833cd8155772-dbus\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:01:56.782481 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-systemd\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.782481 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.782481 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-socket-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.782481 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-ovnkube-config\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.782481 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9jz2\" (UniqueName: \"kubernetes.io/projected/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-kube-api-access-x9jz2\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.782481 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-log-socket\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.782481 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7ea5065d-e1c6-4152-842c-833cd8155772-kubelet-config\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-node-log\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkcbp\" (UniqueName: \"kubernetes.io/projected/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-kube-api-access-qkcbp\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-run-netns\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-cni-bin\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-env-overrides\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-registration-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-sys-fs\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13ece33e-8a7b-418f-aa9e-eaa2fed48273-hosts-file\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ece33e-8a7b-418f-aa9e-eaa2fed48273-tmp-dir\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-etc-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-host-slash\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-cni-netd\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-iptables-alerter-script\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-serviceca\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:01:56.783125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.782895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/596632c1-fb08-4008-ad12-7d399fa3994b-ovn-node-metrics-cert\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.783849 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.783415    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-registration-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.783849 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.783423    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-socket-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.783849 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.783428    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-node-log\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.783849 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.783503    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-host\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:01:56.783849 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.783551    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-var-lib-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.783849 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.783689    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.783849 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.783696    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-run-netns\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784118    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784188    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7ea5065d-e1c6-4152-842c-833cd8155772-dbus\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784233    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-systemd\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784281    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784315    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-cni-bin\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784319    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-systemd-units\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784389    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-kubelet\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:56.784400    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784432    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-slash\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:56.784455    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret podName:7ea5065d-e1c6-4152-842c-833cd8155772 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:57.28443817 +0000 UTC m=+3.110667576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret") pod "global-pull-secret-syncer-qmd98" (UID: "7ea5065d-e1c6-4152-842c-833cd8155772") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784472    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-ovnkube-config\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784478    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-ovnkube-script-lib\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784493    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-device-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784533    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784555    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-log-socket\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784574    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-ovn\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.784700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784608    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-host-slash\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:01:56.785625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784626    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.785625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784634    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7ea5065d-e1c6-4152-842c-833cd8155772-kubelet-config\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:01:56.785625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784667    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-sys-fs\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.785625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784717    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-etc-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.785625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784740    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13ece33e-8a7b-418f-aa9e-eaa2fed48273-hosts-file\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:01:56.785625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784785    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-cni-netd\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.785625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.784925    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ece33e-8a7b-418f-aa9e-eaa2fed48273-tmp-dir\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:01:56.785625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.785194    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-serviceca\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:01:56.785625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.785281    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-iptables-alerter-script\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:01:56.787084 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.787035    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-env-overrides\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.789039 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.788925    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal" event={"ID":"f898674fe0c94d2ff8a087a4cd76c349","Type":"ContainerStarted","Data":"d0fb0377af2f674020219c26fa4a1f42c72cea83140ef2909ceca1f643b17f94"}
Nov 03 16:01:56.789774 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.789749    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/596632c1-fb08-4008-ad12-7d399fa3994b-ovn-node-metrics-cert\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.790330 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.790309    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal" event={"ID":"56a59496f84ad599eee942e332b42fd6","Type":"ContainerStarted","Data":"a1fe67ccc0ffbc19e2e96cbc485098416398466cf149c1884a554c830799fa00"}
Nov 03 16:01:56.796868 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.796843    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svwtn\" (UniqueName: \"kubernetes.io/projected/596632c1-fb08-4008-ad12-7d399fa3994b-kube-api-access-svwtn\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:56.798742 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.798681    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkcbp\" (UniqueName: \"kubernetes.io/projected/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-kube-api-access-qkcbp\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:01:56.798742 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.798691    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9jz2\" (UniqueName: \"kubernetes.io/projected/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-kube-api-access-x9jz2\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.799370 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.799327    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs5sr\" (UniqueName: \"kubernetes.io/projected/13ece33e-8a7b-418f-aa9e-eaa2fed48273-kube-api-access-rs5sr\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:01:56.800250 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.800146    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7s62\" (UniqueName: \"kubernetes.io/projected/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-kube-api-access-v7s62\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:01:56.876040 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.875957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:01:56.882728 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.882704    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:01:56.882877 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.882814    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:01:56.886695 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:56.886669    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d056b1_85f8_48eb_b9bf_aeb4b98a8170.slice/crio-6fee9c50afd4eae2291676cb5f4db803da77aacba92ec62493abd5e3be400be8 WatchSource:0}: Error finding container 6fee9c50afd4eae2291676cb5f4db803da77aacba92ec62493abd5e3be400be8: Status 404 returned error can't find the container with id 6fee9c50afd4eae2291676cb5f4db803da77aacba92ec62493abd5e3be400be8
Nov 03 16:01:56.893163 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:56.893133    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c44c5cc_7fb0_40c7_970b_add5c87c0770.slice/crio-d72a385d5a58c93d91aac71cf9f518bed692fec7095f84aafc6f452ce0837db6 WatchSource:0}: Error finding container d72a385d5a58c93d91aac71cf9f518bed692fec7095f84aafc6f452ce0837db6: Status 404 returned error can't find the container with id d72a385d5a58c93d91aac71cf9f518bed692fec7095f84aafc6f452ce0837db6
Nov 03 16:01:56.907842 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.907822    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n4g6g"
Nov 03 16:01:56.908871 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.908852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:01:56.914675 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.914656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:01:56.920832 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:56.920778    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbf1e42_f480_47a4_8382_6207e2fe7774.slice/crio-6d99e95fb1a77cc650cf6ea4ec19dc3847fe12e58e81982442fe06ff7415c687 WatchSource:0}: Error finding container 6d99e95fb1a77cc650cf6ea4ec19dc3847fe12e58e81982442fe06ff7415c687: Status 404 returned error can't find the container with id 6d99e95fb1a77cc650cf6ea4ec19dc3847fe12e58e81982442fe06ff7415c687
Nov 03 16:01:56.921263 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.921243    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:01:56.923869 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.923841    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:01:56.930717 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.930568    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:01:56.942674 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:01:56.942609    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e65ea9_6a1d_4473_9c23_a5ecf3d4bfa7.slice/crio-5b6a95384b4a90b0adf8bdd0aae714ad41949c944146414374807e9dc4c93a4e WatchSource:0}: Error finding container 5b6a95384b4a90b0adf8bdd0aae714ad41949c944146414374807e9dc4c93a4e: Status 404 returned error can't find the container with id 5b6a95384b4a90b0adf8bdd0aae714ad41949c944146414374807e9dc4c93a4e
Nov 03 16:01:56.952268 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:56.952248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:01:57.187161 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.187083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:01:57.187288 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:57.187230    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:57.187288 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:57.187279    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs podName:a00bade6-34ba-4d87-a740-9727b8372650 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:58.187265109 +0000 UTC m=+4.013494515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs") pod "network-metrics-daemon-sf4cc" (UID: "a00bade6-34ba-4d87-a740-9727b8372650") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:57.288447 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.288410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:01:57.288612 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.288462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4774\" (UniqueName: \"kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774\") pod \"network-check-target-2ht7q\" (UID: \"af63014e-5ed6-4213-8838-529749c931cb\") " pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:01:57.288612 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:57.288574    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:57.288612 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:57.288605    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:01:57.288757 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:57.288623    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:01:57.288757 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:57.288635    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r4774 for pod openshift-network-diagnostics/network-check-target-2ht7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:57.288757 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:57.288653    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret podName:7ea5065d-e1c6-4152-842c-833cd8155772 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:58.288631937 +0000 UTC m=+4.114861359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret") pod "global-pull-secret-syncer-qmd98" (UID: "7ea5065d-e1c6-4152-842c-833cd8155772") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:57.288757 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:57.288680    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774 podName:af63014e-5ed6-4213-8838-529749c931cb nodeName:}" failed. No retries permitted until 2026-11-03 16:01:58.288663631 +0000 UTC m=+4.114893051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4774" (UniqueName: "kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774") pod "network-check-target-2ht7q" (UID: "af63014e-5ed6-4213-8838-529749c931cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:57.663083 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.663022    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:56:56 +0000 UTC" deadline="2028-05-06 02:27:16.201453653 +0000 UTC"
Nov 03 16:01:57.663083 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.663080    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13186h25m18.538376642s"
Nov 03 16:01:57.786537 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.785991    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:01:57.786537 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:57.786126    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:01:57.804410 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.803605    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal" event={"ID":"f898674fe0c94d2ff8a087a4cd76c349","Type":"ContainerStarted","Data":"c6eb742b705ff79bd912977157c9c7eec84a4571818b205700965f88ba846e78"}
Nov 03 16:01:57.809865 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.809730    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4" event={"ID":"596632c1-fb08-4008-ad12-7d399fa3994b","Type":"ContainerStarted","Data":"f42d395a33f328559930baf996d8efe1eb704477cd5e0fc327ac3581fd0bf5cd"}
Nov 03 16:01:57.816250 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.816119    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d" event={"ID":"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0","Type":"ContainerStarted","Data":"78598f65fa6cf2cae3f644a79a9eeb9e7b0eb3c5b355da0b32c5cb9a11c619b3"}
Nov 03 16:01:57.821029 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.820956    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsg95" event={"ID":"c0652c5d-5440-4b54-9220-fbf77c06ec2f","Type":"ContainerStarted","Data":"1b60c8b03a2166e60967970a0a532a3e32e619e8c11a006d43b2aed5cb1f2e67"}
Nov 03 16:01:57.824206 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.824146    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j5r6g" event={"ID":"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998","Type":"ContainerStarted","Data":"0154a7620c880551c3e2fd70a07c4c1a3737e2d53f1bf32fd621f3996dd989e7"}
Nov 03 16:01:57.828029 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.827934    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n4g6g" event={"ID":"1dbf1e42-f480-47a4-8382-6207e2fe7774","Type":"ContainerStarted","Data":"6d99e95fb1a77cc650cf6ea4ec19dc3847fe12e58e81982442fe06ff7415c687"}
Nov 03 16:01:57.830073 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.830022    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rlgrt" event={"ID":"1c44c5cc-7fb0-40c7-970b-add5c87c0770","Type":"ContainerStarted","Data":"d72a385d5a58c93d91aac71cf9f518bed692fec7095f84aafc6f452ce0837db6"}
Nov 03 16:01:57.831541 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.831488    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hdgx7" event={"ID":"13d056b1-85f8-48eb-b9bf-aeb4b98a8170","Type":"ContainerStarted","Data":"6fee9c50afd4eae2291676cb5f4db803da77aacba92ec62493abd5e3be400be8"}
Nov 03 16:01:57.833039 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.832986    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5flz8" event={"ID":"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7","Type":"ContainerStarted","Data":"5b6a95384b4a90b0adf8bdd0aae714ad41949c944146414374807e9dc4c93a4e"}
Nov 03 16:01:57.834637 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:57.834588    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7f6nb" event={"ID":"13ece33e-8a7b-418f-aa9e-eaa2fed48273","Type":"ContainerStarted","Data":"7da326ffd374624f716d47a5caad64927991919e4cfc64e5a6377275609f4107"}
Nov 03 16:01:58.195954 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:58.195601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:01:58.196063 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:58.195737    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:58.196123 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:58.196101    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs podName:a00bade6-34ba-4d87-a740-9727b8372650 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:00.196078262 +0000 UTC m=+6.022307667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs") pod "network-metrics-daemon-sf4cc" (UID: "a00bade6-34ba-4d87-a740-9727b8372650") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:58.298104 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:58.297115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:01:58.298104 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:58.297191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4774\" (UniqueName: \"kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774\") pod \"network-check-target-2ht7q\" (UID: \"af63014e-5ed6-4213-8838-529749c931cb\") " pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:01:58.298104 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:58.297350    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:01:58.298104 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:58.297373    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:01:58.298104 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:58.297387    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r4774 for pod openshift-network-diagnostics/network-check-target-2ht7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:58.298104 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:58.297521    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774 podName:af63014e-5ed6-4213-8838-529749c931cb nodeName:}" failed. No retries permitted until 2026-11-03 16:02:00.297501549 +0000 UTC m=+6.123730959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4774" (UniqueName: "kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774") pod "network-check-target-2ht7q" (UID: "af63014e-5ed6-4213-8838-529749c931cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:58.298104 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:58.298011    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:58.298104 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:58.298066    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret podName:7ea5065d-e1c6-4152-842c-833cd8155772 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:00.298050972 +0000 UTC m=+6.124280381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret") pod "global-pull-secret-syncer-qmd98" (UID: "7ea5065d-e1c6-4152-842c-833cd8155772") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:58.503923 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:58.503885    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:01:58.665647 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:58.665601    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:56:56 +0000 UTC" deadline="2028-05-03 21:28:14.954531524 +0000 UTC"
Nov 03 16:01:58.665647 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:58.665644    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13133h26m16.288891943s"
Nov 03 16:01:58.789222 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:58.789120    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:01:58.789437 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:58.789266    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:01:58.789752 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:58.789734    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:01:58.789873 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:58.789852    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:01:58.839993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:58.839960    2577 generic.go:356] "Generic (PLEG): container finished" podID="56a59496f84ad599eee942e332b42fd6" containerID="6625f2354d9716f3a80855252f360d1b293a181169975be43f7f6c278e27c089" exitCode=0
Nov 03 16:01:58.840549 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:58.840521    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal" event={"ID":"56a59496f84ad599eee942e332b42fd6","Type":"ContainerDied","Data":"6625f2354d9716f3a80855252f360d1b293a181169975be43f7f6c278e27c089"}
Nov 03 16:01:58.861576 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:58.861516    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal" podStartSLOduration=3.86149876 podStartE2EDuration="3.86149876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:57.82298896 +0000 UTC m=+3.649218385" watchObservedRunningTime="2026-11-03 16:01:58.86149876 +0000 UTC m=+4.687728189"
Nov 03 16:01:59.786617 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:59.786582    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:01:59.787022 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:01:59.786711    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:01:59.857436 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:01:59.857362    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal" event={"ID":"56a59496f84ad599eee942e332b42fd6","Type":"ContainerStarted","Data":"341b5de13274a7f8c536949691d893d4f0bfe163e7375abc62e87514476b0331"}
Nov 03 16:02:00.213915 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:00.213784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:00.214074 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:00.213964    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:02:00.214074 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:00.214027    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs podName:a00bade6-34ba-4d87-a740-9727b8372650 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:04.214007601 +0000 UTC m=+10.040237007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs") pod "network-metrics-daemon-sf4cc" (UID: "a00bade6-34ba-4d87-a740-9727b8372650") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:02:00.314655 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:00.314613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:00.314787 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:00.314690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4774\" (UniqueName: \"kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774\") pod \"network-check-target-2ht7q\" (UID: \"af63014e-5ed6-4213-8838-529749c931cb\") " pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:00.314956 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:00.314880    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:02:00.315058 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:00.315043    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:02:00.315253 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:00.315148    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret podName:7ea5065d-e1c6-4152-842c-833cd8155772 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:04.315128765 +0000 UTC m=+10.141358183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret") pod "global-pull-secret-syncer-qmd98" (UID: "7ea5065d-e1c6-4152-842c-833cd8155772") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:02:00.315770 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:00.315719    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:02:00.315770 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:00.315752    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r4774 for pod openshift-network-diagnostics/network-check-target-2ht7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:02:00.315917 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:00.315820    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774 podName:af63014e-5ed6-4213-8838-529749c931cb nodeName:}" failed. No retries permitted until 2026-11-03 16:02:04.315789094 +0000 UTC m=+10.142018517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4774" (UniqueName: "kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774") pod "network-check-target-2ht7q" (UID: "af63014e-5ed6-4213-8838-529749c931cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:02:00.788328 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:00.788272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:00.788859 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:00.788405    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:00.788859 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:00.788484    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:00.788859 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:00.788570    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:00.871342 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:00.866992    2577 generic.go:356] "Generic (PLEG): container finished" podID="c0652c5d-5440-4b54-9220-fbf77c06ec2f" containerID="03d464390d235d6ca924e4ea4ac28db3de756236b7c06f91c615eb5dd8bbe587" exitCode=0
Nov 03 16:02:00.871342 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:00.868079    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsg95" event={"ID":"c0652c5d-5440-4b54-9220-fbf77c06ec2f","Type":"ContainerDied","Data":"03d464390d235d6ca924e4ea4ac28db3de756236b7c06f91c615eb5dd8bbe587"}
Nov 03 16:02:01.021634 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:01.021528    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal" podStartSLOduration=6.021508164 podStartE2EDuration="6.021508164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:59.877417782 +0000 UTC m=+5.703647210" watchObservedRunningTime="2026-11-03 16:02:01.021508164 +0000 UTC m=+6.847737588"
Nov 03 16:02:01.785205 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:01.785169    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:01.785402 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:01.785315    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:01.871961 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:01.871921    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hdgx7" event={"ID":"13d056b1-85f8-48eb-b9bf-aeb4b98a8170","Type":"ContainerStarted","Data":"70531fa30ad68bb6b823787890e16130bd9f20c15c742eb2788e5ce1ee7d9180"}
Nov 03 16:02:02.788010 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:02.787978    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:02.788161 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:02.788104    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:02.788513 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:02.788491    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:02.788617 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:02.788594    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:02.876677 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:02.876637    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d" event={"ID":"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0","Type":"ContainerStarted","Data":"7f3334985aa2c419c8862cb3d2287d17440660305efeae0efec10e7f83450a02"}
Nov 03 16:02:02.880005 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:02.879976    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j5r6g" event={"ID":"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998","Type":"ContainerStarted","Data":"bd33d73c78339eb673368586e8bf9bc97895acf442ca39a1f93573d0c4b32c62"}
Nov 03 16:02:02.901629 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:02.901574    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hdgx7" podStartSLOduration=5.085790883 podStartE2EDuration="8.901561287s" totalImagesPullingTime="3.815770404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:01.907443952 +0000 UTC m=+7.733673381" watchObservedRunningTime="2026-11-03 16:02:02.901561287 +0000 UTC m=+8.727790746"
Nov 03 16:02:03.258771 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:03.258731    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:02:03.259266 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:03.259227    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:02:03.278526 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:03.278467    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j5r6g" podStartSLOduration=4.324154715 podStartE2EDuration="9.278449048s" totalImagesPullingTime="4.954294333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:02.902392554 +0000 UTC m=+8.728621981" watchObservedRunningTime="2026-11-03 16:02:03.278449048 +0000 UTC m=+9.104678480"
Nov 03 16:02:03.785005 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:03.784971    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:03.785193 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:03.785090    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:04.250473 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:04.250442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:04.250991 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:04.250599    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:02:04.250991 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:04.250652    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs podName:a00bade6-34ba-4d87-a740-9727b8372650 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:12.250634939 +0000 UTC m=+18.076864348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs") pod "network-metrics-daemon-sf4cc" (UID: "a00bade6-34ba-4d87-a740-9727b8372650") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:02:04.268815 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:04.268637    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 16:02:04.352174 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:04.351421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4774\" (UniqueName: \"kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774\") pod \"network-check-target-2ht7q\" (UID: \"af63014e-5ed6-4213-8838-529749c931cb\") " pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:04.352174 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:04.351524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:04.352174 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:04.351686    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:02:04.352174 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:04.351755    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:02:04.352174 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:04.351772    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret podName:7ea5065d-e1c6-4152-842c-833cd8155772 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:12.351750467 +0000 UTC m=+18.177979897 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret") pod "global-pull-secret-syncer-qmd98" (UID: "7ea5065d-e1c6-4152-842c-833cd8155772") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:02:04.352174 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:04.351782    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:02:04.352174 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:04.351814    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r4774 for pod openshift-network-diagnostics/network-check-target-2ht7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:02:04.352174 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:04.351865    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774 podName:af63014e-5ed6-4213-8838-529749c931cb nodeName:}" failed. No retries permitted until 2026-11-03 16:02:12.351849256 +0000 UTC m=+18.178078676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4774" (UniqueName: "kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774") pod "network-check-target-2ht7q" (UID: "af63014e-5ed6-4213-8838-529749c931cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:02:04.635372 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:04.635180    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:02:04.268658613Z","UUID":"f548d806-1875-4067-9384-3e02a785f569","Handler":null,"Name":"","Endpoint":""}
Nov 03 16:02:04.636764 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:04.636746    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 16:02:04.636875 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:04.636772    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 16:02:04.788816 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:04.786894    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:04.788816 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:04.787027    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:04.788816 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:04.787217    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:04.788816 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:04.787312    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:04.885677 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:04.885592    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n4g6g" event={"ID":"1dbf1e42-f480-47a4-8382-6207e2fe7774","Type":"ContainerStarted","Data":"378d6959ca7142f99cb6bdf3bc99c64c332093928da58851c831830cb3599ffb"}
Nov 03 16:02:04.889239 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:04.889208    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7f6nb" event={"ID":"13ece33e-8a7b-418f-aa9e-eaa2fed48273","Type":"ContainerStarted","Data":"baa64eee08cd58f319d5d40e74a9485261f1647a73a1875e27a51e6335df9bdd"}
Nov 03 16:02:04.892490 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:04.892465    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d" event={"ID":"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0","Type":"ContainerStarted","Data":"3d90069561e11d7e7d815d8ca3ce89518f1b560a18150fd9ffd21d46df46fab5"}
Nov 03 16:02:04.892588 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:04.892512    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 16:02:04.909438 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:04.909384    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-n4g6g" podStartSLOduration=3.467512129 podStartE2EDuration="10.909366795s" totalImagesPullingTime="7.441854666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:04.908720859 +0000 UTC m=+10.734950287" watchObservedRunningTime="2026-11-03 16:02:04.909366795 +0000 UTC m=+10.735596225"
Nov 03 16:02:04.927839 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:04.927767    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7f6nb" podStartSLOduration=3.199207449 podStartE2EDuration="10.927750905s" totalImagesPullingTime="7.728543456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:04.927683522 +0000 UTC m=+10.753912948" watchObservedRunningTime="2026-11-03 16:02:04.927750905 +0000 UTC m=+10.753980333"
Nov 03 16:02:05.785058 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:05.785028    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:05.785497 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:05.785159    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:05.897362 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:05.897279    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d" event={"ID":"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0","Type":"ContainerStarted","Data":"57819c5bd0104620ea0612348715c71aa8f8fb1f4144dad152a414161866d934"}
Nov 03 16:02:05.899402 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:05.899366    2577 generic.go:356] "Generic (PLEG): container finished" podID="c0652c5d-5440-4b54-9220-fbf77c06ec2f" containerID="001e57073919f5d0ec1cf540e3ad856fcaf84e42fb0f1b76e6d253006bc82b60" exitCode=0
Nov 03 16:02:05.899536 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:05.899449    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsg95" event={"ID":"c0652c5d-5440-4b54-9220-fbf77c06ec2f","Type":"ContainerDied","Data":"001e57073919f5d0ec1cf540e3ad856fcaf84e42fb0f1b76e6d253006bc82b60"}
Nov 03 16:02:05.958304 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:05.958248    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d" podStartSLOduration=3.903929215 podStartE2EDuration="11.958235232s" totalImagesPullingTime="8.054306017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:05.957871281 +0000 UTC m=+11.784100709" watchObservedRunningTime="2026-11-03 16:02:05.958235232 +0000 UTC m=+11.784464659"
Nov 03 16:02:06.785191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:06.784946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:06.785730 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:06.784946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:06.785730 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:06.785259    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:06.785730 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:06.785307    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:06.904639 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:06.904588    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5flz8" event={"ID":"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7","Type":"ContainerStarted","Data":"729bbb2c6a4eca6bbb361ff98fab0ce7229ccc5204e24d062845846669280d54"}
Nov 03 16:02:06.925519 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:06.925473    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5flz8" podStartSLOduration=4.706651194 podStartE2EDuration="12.925459602s" totalImagesPullingTime="8.218808408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:06.925087802 +0000 UTC m=+12.751317230" watchObservedRunningTime="2026-11-03 16:02:06.925459602 +0000 UTC m=+12.751689031"
Nov 03 16:02:07.343302 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:07.343252    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:07.560482 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:07.560401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:07.785075 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:07.785049    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:07.785251 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:07.785177    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:07.908979 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:07.908886    2577 generic.go:356] "Generic (PLEG): container finished" podID="c0652c5d-5440-4b54-9220-fbf77c06ec2f" containerID="ca19d25eab2c1acbbf021e4f363d909352d953e649036fa2aecfc6ade25b27fd" exitCode=0
Nov 03 16:02:07.909110 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:07.908975    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsg95" event={"ID":"c0652c5d-5440-4b54-9220-fbf77c06ec2f","Type":"ContainerDied","Data":"ca19d25eab2c1acbbf021e4f363d909352d953e649036fa2aecfc6ade25b27fd"}
Nov 03 16:02:07.910739 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:07.910712    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rlgrt" event={"ID":"1c44c5cc-7fb0-40c7-970b-add5c87c0770","Type":"ContainerStarted","Data":"d9887ddb08b71e40f696b041f0c1647b368b297f6d556bdf4c61dddc3907ba30"}
Nov 03 16:02:07.966023 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:07.965974    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rlgrt" podStartSLOduration=3.693721671 podStartE2EDuration="13.965959218s" totalImagesPullingTime="10.272237547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:07.965374437 +0000 UTC m=+13.791603866" watchObservedRunningTime="2026-11-03 16:02:07.965959218 +0000 UTC m=+13.792188647"
Nov 03 16:02:08.785598 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:08.785569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:08.786093 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:08.785696    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:08.786093 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:08.785734    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:08.786093 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:08.785827    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:08.889656 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:08.889623    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:08.916007 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:08.915975    2577 generic.go:356] "Generic (PLEG): container finished" podID="c0652c5d-5440-4b54-9220-fbf77c06ec2f" containerID="8bfa8eb45fbe503006ce52cfc6604f851bb4ad45c0fe30a7c2de3e68a10120b2" exitCode=0
Nov 03 16:02:08.916117 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:08.916056    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsg95" event={"ID":"c0652c5d-5440-4b54-9220-fbf77c06ec2f","Type":"ContainerDied","Data":"8bfa8eb45fbe503006ce52cfc6604f851bb4ad45c0fe30a7c2de3e68a10120b2"}
Nov 03 16:02:09.042256 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:09.042168    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:09.784948 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:09.784910    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:09.785096 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:09.785018    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:10.785812 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:10.785625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:10.786275 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:10.785650    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:10.786275 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:10.785932    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:10.786275 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:10.785985    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:10.843882 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:10.843851    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:10.923370 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:10.923343    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsg95" event={"ID":"c0652c5d-5440-4b54-9220-fbf77c06ec2f","Type":"ContainerStarted","Data":"92d40c6fab91fdaca94f6d2f7bf00c6937da427a991ccd99e9e8dba0737b160d"}
Nov 03 16:02:11.010031 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:11.009948    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:11.380480 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:11.380401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:11.529218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:11.529186    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:11.785331 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:11.785305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:11.785471 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:11.785399    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:11.928355 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:11.928321    2577 generic.go:356] "Generic (PLEG): container finished" podID="c0652c5d-5440-4b54-9220-fbf77c06ec2f" containerID="92d40c6fab91fdaca94f6d2f7bf00c6937da427a991ccd99e9e8dba0737b160d" exitCode=0
Nov 03 16:02:11.928705 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:11.928418    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsg95" event={"ID":"c0652c5d-5440-4b54-9220-fbf77c06ec2f","Type":"ContainerDied","Data":"92d40c6fab91fdaca94f6d2f7bf00c6937da427a991ccd99e9e8dba0737b160d"}
Nov 03 16:02:11.931504 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:11.931410    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4" event={"ID":"596632c1-fb08-4008-ad12-7d399fa3994b","Type":"ContainerStarted","Data":"678142ec7b32826acc547aede49a682c8bf40b8bd617e51c8468edc5dad76000"}
Nov 03 16:02:11.931504 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:11.931469    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4" event={"ID":"596632c1-fb08-4008-ad12-7d399fa3994b","Type":"ContainerStarted","Data":"5058adfe2e59d9aba0d1158ff79589e8c3e4c2e43b97b2a04537c3d68e4a8a57"}
Nov 03 16:02:11.931504 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:11.931498    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4" event={"ID":"596632c1-fb08-4008-ad12-7d399fa3994b","Type":"ContainerStarted","Data":"d035025cb24dad1df8c69a08ed5e7d8a429b898ef2226cc00fc7f442532c8750"}
Nov 03 16:02:11.931614 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:11.931525    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4" event={"ID":"596632c1-fb08-4008-ad12-7d399fa3994b","Type":"ContainerStarted","Data":"1254a4d116119c9bf33c9a0456f7bde468d57361744d26626960f8511b3e294e"}
Nov 03 16:02:11.931614 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:11.931551    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4" event={"ID":"596632c1-fb08-4008-ad12-7d399fa3994b","Type":"ContainerStarted","Data":"078780f221e686a44316bbb1783f4fbb82d42a032a3b5e805bbd645ca1c60692"}
Nov 03 16:02:12.323344 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:12.323134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:12.323344 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:12.323267    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:02:12.323500 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:12.323370    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs podName:a00bade6-34ba-4d87-a740-9727b8372650 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:28.323356723 +0000 UTC m=+34.149586129 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs") pod "network-metrics-daemon-sf4cc" (UID: "a00bade6-34ba-4d87-a740-9727b8372650") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:02:12.423633 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:12.423606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:12.423784 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:12.423655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4774\" (UniqueName: \"kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774\") pod \"network-check-target-2ht7q\" (UID: \"af63014e-5ed6-4213-8838-529749c931cb\") " pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:12.423784 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:12.423741    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:02:12.423784 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:12.423777    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:02:12.423957 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:12.423818    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:02:12.423957 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:12.423823    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret podName:7ea5065d-e1c6-4152-842c-833cd8155772 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:28.423783337 +0000 UTC m=+34.250012741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret") pod "global-pull-secret-syncer-qmd98" (UID: "7ea5065d-e1c6-4152-842c-833cd8155772") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:02:12.423957 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:12.423829    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r4774 for pod openshift-network-diagnostics/network-check-target-2ht7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:02:12.423957 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:12.423882    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774 podName:af63014e-5ed6-4213-8838-529749c931cb nodeName:}" failed. No retries permitted until 2026-11-03 16:02:28.423866661 +0000 UTC m=+34.250096072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4774" (UniqueName: "kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774") pod "network-check-target-2ht7q" (UID: "af63014e-5ed6-4213-8838-529749c931cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:02:12.785919 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:12.785850    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:12.786069 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:12.785851    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:12.786069 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:12.785992    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:12.786069 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:12.786047    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:12.935847 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:12.935816    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4" event={"ID":"596632c1-fb08-4008-ad12-7d399fa3994b","Type":"ContainerStarted","Data":"afca4d09785e264fa58426e3429027edc1eca812a81469da0db06cea39f4d758"}
Nov 03 16:02:12.938700 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:12.938679    2577 generic.go:356] "Generic (PLEG): container finished" podID="c0652c5d-5440-4b54-9220-fbf77c06ec2f" containerID="a0a69fe79ddc49c8aa9e945e8486632e5f20bde05a4c326c7f52df4a60033a90" exitCode=0
Nov 03 16:02:12.938790 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:12.938712    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsg95" event={"ID":"c0652c5d-5440-4b54-9220-fbf77c06ec2f","Type":"ContainerDied","Data":"a0a69fe79ddc49c8aa9e945e8486632e5f20bde05a4c326c7f52df4a60033a90"}
Nov 03 16:02:13.168283 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:13.168197    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:13.340148 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:13.340120    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:13.612885 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:13.612807    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:02:13.613040 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:13.612927    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 16:02:13.613323 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:13.613309    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:02:13.783359 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:13.783322    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:13.785115 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:13.785098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:13.785208 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:13.785190    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:13.943646 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:13.943578    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vsg95" event={"ID":"c0652c5d-5440-4b54-9220-fbf77c06ec2f","Type":"ContainerStarted","Data":"856e7c1ed2cbb2849c3916292e6b6d63ec5de37ebad6575247d1c3bdaf204bb7"}
Nov 03 16:02:13.978939 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:13.978893    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vsg95" podStartSLOduration=9.247166014 podStartE2EDuration="19.978881997s" totalImagesPullingTime="10.731715983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:13.978779426 +0000 UTC m=+19.805008852" watchObservedRunningTime="2026-11-03 16:02:13.978881997 +0000 UTC m=+19.805111424"
Nov 03 16:02:13.991739 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:13.991713    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:14.205841 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:14.205766    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:14.414844 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:14.414780    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:14.694939 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:14.694853    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:14.786965 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:14.786937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:14.787097 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:14.787058    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:14.787097 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:14.787092    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:14.787184 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:14.787168    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:14.857697 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:14.857665    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:14.948862 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:14.948788    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4" event={"ID":"596632c1-fb08-4008-ad12-7d399fa3994b","Type":"ContainerStarted","Data":"ef365b3eab39092d2ed9b826bd9c91e43724f8c0d8abeda1cbeee01390e834d3"}
Nov 03 16:02:15.056785 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:15.056753    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:15.214374 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:15.214296    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:15.550257 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:15.550175    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:15.785863 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:15.785838    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:15.785999 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:15.785923    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:15.883702 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:15.883644    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:16.061546 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:16.061513    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:16.290491 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:16.290455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:16.785407 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:16.785376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:16.785407 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:16.785409    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:16.785595 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:16.785486    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:16.785632 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:16.785598    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:16.791705 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:16.791683    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:16.955437 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:16.955402    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4" event={"ID":"596632c1-fb08-4008-ad12-7d399fa3994b","Type":"ContainerStarted","Data":"9882983c909cd002b379350abd25e24750902c7df53734962d0786eec9c501a5"}
Nov 03 16:02:16.955664 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:16.955644    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:02:16.955736 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:16.955715    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:02:16.971086 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:16.971066    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:02:16.997864 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:16.997826    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4" podStartSLOduration=8.337765223 podStartE2EDuration="22.997813961s" totalImagesPullingTime="14.660048738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:16.997328542 +0000 UTC m=+22.823557969" watchObservedRunningTime="2026-11-03 16:02:16.997813961 +0000 UTC m=+22.824043382"
Nov 03 16:02:17.196110 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:17.196034    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:17.360838 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:17.360812    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7f6nb_13ece33e-8a7b-418f-aa9e-eaa2fed48273/dns-node-resolver/0.log"
Nov 03 16:02:17.592588 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:17.592508    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:17.785619 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:17.785594    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:17.785731 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:17.785680    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:17.958124 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:17.958057    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:02:17.973469 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:17.973445    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:02:17.992468 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:17.992441    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:18.405345 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:18.405277    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:18.557653 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:18.557589    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:18.562095 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:18.562075    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j5r6g_e3b0f6f7-b58c-49a1-aa66-57cee4ae2998/node-ca/0.log"
Nov 03 16:02:18.719083 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:18.719006    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:18.785925 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:18.785898    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:18.786055 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:18.785926    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:18.786055 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:18.786007    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:18.786133 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:18.786121    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:18.877088 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:18.877059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:19.180411 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:19.180338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:19.346319 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:19.346287    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:19.656701 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:19.656613    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:19.784981 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:19.784955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:19.785136 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:19.785048    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:19.818217 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:19.818181    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:19.993632 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:19.993407    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:20.395398 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:20.395319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:20.785647 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:20.785576    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:20.786039 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:20.785681    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:20.786039 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:20.785751    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:20.786039 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:20.785849    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:20.791933 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:20.791912    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:21.184394 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:21.184307    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:21.591623 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:21.591546    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:21.785162 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:21.785137    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:21.785294 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:21.785221    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:21.984289 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:21.984211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:22.385261 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:22.385182    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:22.785748 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:22.785677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:22.785748 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:22.785687    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:22.785931 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:22.785776    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:22.785931 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:22.785896    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:22.790653 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:22.790634    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:23.188681 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:23.188581    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:23.588577 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:23.588491    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:23.785226 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:23.785196    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:23.785372 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:23.785282    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:23.890753 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:23.890684    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:24.041097 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:24.041060    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:24.391018 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:24.390997    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:24.785763 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:24.785694    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:24.785954 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:24.785841    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:24.785954 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:24.785903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:24.786069 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:24.786008    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:25.784976 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:25.784953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:25.785344 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:25.785044    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:25.805148 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:25.805110    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:25.962021 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:25.961996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:26.119742 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:26.119666    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:26.279456 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:26.279433    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:26.430188 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:26.430120    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:26.785358 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:26.785287    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:26.785358 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:26.785293    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:26.785773 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:26.785447    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:26.785773 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:26.785483    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:26.786655 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:26.786630    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:27.612652 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:27.612618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:27.785235 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:27.785205    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:27.785451 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:27.785297    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:28.331831 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:28.331786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:28.331994 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:28.331902    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:02:28.331994 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:28.331953    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs podName:a00bade6-34ba-4d87-a740-9727b8372650 nodeName:}" failed. No retries permitted until 2026-11-03 16:03:00.331940034 +0000 UTC m=+66.158169440 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs") pod "network-metrics-daemon-sf4cc" (UID: "a00bade6-34ba-4d87-a740-9727b8372650") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:02:28.432432 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:28.432398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:28.432600 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:28.432447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4774\" (UniqueName: \"kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774\") pod \"network-check-target-2ht7q\" (UID: \"af63014e-5ed6-4213-8838-529749c931cb\") " pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:28.432600 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:28.432530    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:02:28.432600 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:28.432582    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret podName:7ea5065d-e1c6-4152-842c-833cd8155772 nodeName:}" failed. No retries permitted until 2026-11-03 16:03:00.432568634 +0000 UTC m=+66.258798043 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret") pod "global-pull-secret-syncer-qmd98" (UID: "7ea5065d-e1c6-4152-842c-833cd8155772") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:02:28.432600 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:28.432585    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:02:28.432600 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:28.432604    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:02:28.432832 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:28.432615    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r4774 for pod openshift-network-diagnostics/network-check-target-2ht7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:02:28.432832 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:28.432661    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774 podName:af63014e-5ed6-4213-8838-529749c931cb nodeName:}" failed. No retries permitted until 2026-11-03 16:03:00.432644008 +0000 UTC m=+66.258873415 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4774" (UniqueName: "kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774") pod "network-check-target-2ht7q" (UID: "af63014e-5ed6-4213-8838-529749c931cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:02:28.785432 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:28.785407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:28.785595 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:28.785407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:28.785595 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:28.785508    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:28.785938 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:28.785598    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:29.785730 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:29.785701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:29.786145 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:29.785809    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:30.785170 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:30.785143    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:30.785336 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:30.785236    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:30.785336 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:30.785311    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:30.785408 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:30.785390    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:31.785771 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:31.785742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:31.786148 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:31.785867    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:32.785344 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:32.785318    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:32.785516 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:32.785484    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:32.785560 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:32.785520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:32.785632 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:32.785615    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:33.310667 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:33.310637    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:33.561068 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:33.560989    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:33.785207 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:33.785177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:33.785354 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:33.785270    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:33.787954 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:33.787929    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:34.038452 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:34.038412    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:34.196963 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:34.196932    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:34.352139 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:34.352056    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:34.591034 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:34.591003    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:34.787474 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:34.787443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:34.787616 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:34.787565    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:34.787616 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:34.787574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:34.787713 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:34.787677    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:34.990836 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:34.990743    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2ht7q"]
Nov 03 16:02:34.990970 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:34.990884    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:34.991018 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:34.990989    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:34.993142 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:34.993118    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qmd98"]
Nov 03 16:02:34.993263 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:34.993193    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:34.993316 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:34.993268    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:35.023753 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:35.023725    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sf4cc"]
Nov 03 16:02:35.023918 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:35.023853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:35.023977 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:35.023941    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:35.166941 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:35.166889    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:35.601078 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:35.601041    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:35.780909 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:35.780874    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:36.283567 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:36.283543    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7f6nb_13ece33e-8a7b-418f-aa9e-eaa2fed48273/dns-node-resolver/0.log"
Nov 03 16:02:36.560103 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:36.560073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:36.714543 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:36.714509    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:36.785698 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:36.785665    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:36.785878 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:36.785670    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:36.785878 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:36.785783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:36.785878 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:36.785777    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:36.785999 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:36.785879    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:36.785999 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:36.785941    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:36.953625 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:36.953556    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:37.106245 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:37.106210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:37.263212 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:37.263126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:37.418722 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:37.418693    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:37.683452 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:37.683423    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j5r6g_e3b0f6f7-b58c-49a1-aa66-57cee4ae2998/node-ca/0.log"
Nov 03 16:02:38.785337 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:38.785138    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:38.785669 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:38.785194    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:38.785669 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:38.785443    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ht7q" podUID="af63014e-5ed6-4213-8838-529749c931cb"
Nov 03 16:02:38.785669 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:38.785234    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:38.785669 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:38.785508    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sf4cc" podUID="a00bade6-34ba-4d87-a740-9727b8372650"
Nov 03 16:02:38.785669 ip-10-0-137-159 kubenswrapper[2577]: E1103 16:02:38.785586    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmd98" podUID="7ea5065d-e1c6-4152-842c-833cd8155772"
Nov 03 16:02:38.891076 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:38.891043    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:38.995144 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:38.995118    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeReady"
Nov 03 16:02:38.995274 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:38.995225    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 16:02:39.050688 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.050665    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hdzxp"]
Nov 03 16:02:39.052489 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.052472    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.053661 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.053637    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kgdhx"]
Nov 03 16:02:39.055426 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.055410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:39.057011 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.056990    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 16:02:39.057125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.057057    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6bb2h\""
Nov 03 16:02:39.057125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.057057    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 16:02:39.057125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.057095    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 16:02:39.057125 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.057117    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 16:02:39.058158 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.058138    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:39.085429 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.085411    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hdzxp"]
Nov 03 16:02:39.091136 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.091117    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kgdhx"]
Nov 03 16:02:39.157066 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.157042    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8szjr\""
Nov 03 16:02:39.157066 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.157054    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 16:02:39.157066 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.157057    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 16:02:39.159088 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.159070    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8vvpp"]
Nov 03 16:02:39.160590 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.160578    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8vvpp"
Nov 03 16:02:39.171532 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.171510    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8vvpp"]
Nov 03 16:02:39.209528 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.209508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krj9t\" (UniqueName: \"kubernetes.io/projected/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-kube-api-access-krj9t\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:39.209635 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.209551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61353437-1060-48e2-89f0-b51beb1fdb60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.209635 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.209574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-config-volume\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:39.209635 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.209622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-metrics-tls\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:39.209751 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.209718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61353437-1060-48e2-89f0-b51beb1fdb60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.209816 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.209763    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-tmp-dir\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:39.209816 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.209780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61353437-1060-48e2-89f0-b51beb1fdb60-data-volume\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.209897 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.209813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66f87\" (UniqueName: \"kubernetes.io/projected/61353437-1060-48e2-89f0-b51beb1fdb60-kube-api-access-66f87\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.209897 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.209879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61353437-1060-48e2-89f0-b51beb1fdb60-crio-socket\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.259905 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.259880    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jk68z\""
Nov 03 16:02:39.259978 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.259945    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 16:02:39.260191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.260173    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 16:02:39.260191 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.260187    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 16:02:39.310458 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.310402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24wql\" (UniqueName: \"kubernetes.io/projected/a9f7db5a-c0f7-480e-a19f-8b77e7285af2-kube-api-access-24wql\") pod \"ingress-canary-8vvpp\" (UID: \"a9f7db5a-c0f7-480e-a19f-8b77e7285af2\") " pod="openshift-ingress-canary/ingress-canary-8vvpp"
Nov 03 16:02:39.310458 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.310436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-config-volume\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:39.310591 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.310462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-metrics-tls\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:39.310591 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.310522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61353437-1060-48e2-89f0-b51beb1fdb60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.310591 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.310561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-tmp-dir\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:39.310591 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.310576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61353437-1060-48e2-89f0-b51beb1fdb60-data-volume\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.310771 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.310593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66f87\" (UniqueName: \"kubernetes.io/projected/61353437-1060-48e2-89f0-b51beb1fdb60-kube-api-access-66f87\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.310771 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.310611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61353437-1060-48e2-89f0-b51beb1fdb60-crio-socket\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.310771 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.310630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9f7db5a-c0f7-480e-a19f-8b77e7285af2-cert\") pod \"ingress-canary-8vvpp\" (UID: \"a9f7db5a-c0f7-480e-a19f-8b77e7285af2\") " pod="openshift-ingress-canary/ingress-canary-8vvpp"
Nov 03 16:02:39.310771 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.310667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krj9t\" (UniqueName: \"kubernetes.io/projected/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-kube-api-access-krj9t\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:39.310771 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.310711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61353437-1060-48e2-89f0-b51beb1fdb60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.311094 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.311066    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-tmp-dir\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:39.311204 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.311119    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61353437-1060-48e2-89f0-b51beb1fdb60-data-volume\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.311264 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.311217    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61353437-1060-48e2-89f0-b51beb1fdb60-crio-socket\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.311318 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.311275    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-config-volume\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:39.311318 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.311290    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61353437-1060-48e2-89f0-b51beb1fdb60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.314553 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.314528    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-metrics-tls\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:39.314553 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.314540    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61353437-1060-48e2-89f0-b51beb1fdb60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.325769 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.325743    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krj9t\" (UniqueName: \"kubernetes.io/projected/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-kube-api-access-krj9t\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:39.326212 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.326194    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66f87\" (UniqueName: \"kubernetes.io/projected/61353437-1060-48e2-89f0-b51beb1fdb60-kube-api-access-66f87\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.367280 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.367258    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:02:39.370424 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.370404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:39.411559 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.411024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24wql\" (UniqueName: \"kubernetes.io/projected/a9f7db5a-c0f7-480e-a19f-8b77e7285af2-kube-api-access-24wql\") pod \"ingress-canary-8vvpp\" (UID: \"a9f7db5a-c0f7-480e-a19f-8b77e7285af2\") " pod="openshift-ingress-canary/ingress-canary-8vvpp"
Nov 03 16:02:39.411559 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.411100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9f7db5a-c0f7-480e-a19f-8b77e7285af2-cert\") pod \"ingress-canary-8vvpp\" (UID: \"a9f7db5a-c0f7-480e-a19f-8b77e7285af2\") " pod="openshift-ingress-canary/ingress-canary-8vvpp"
Nov 03 16:02:39.413732 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.413709    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9f7db5a-c0f7-480e-a19f-8b77e7285af2-cert\") pod \"ingress-canary-8vvpp\" (UID: \"a9f7db5a-c0f7-480e-a19f-8b77e7285af2\") " pod="openshift-ingress-canary/ingress-canary-8vvpp"
Nov 03 16:02:39.429946 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.428584    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24wql\" (UniqueName: \"kubernetes.io/projected/a9f7db5a-c0f7-480e-a19f-8b77e7285af2-kube-api-access-24wql\") pod \"ingress-canary-8vvpp\" (UID: \"a9f7db5a-c0f7-480e-a19f-8b77e7285af2\") " pod="openshift-ingress-canary/ingress-canary-8vvpp"
Nov 03 16:02:39.469410 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.469381    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8vvpp"
Nov 03 16:02:39.661270 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.661239    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kgdhx"]
Nov 03 16:02:39.662060 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.662040    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hdzxp"]
Nov 03 16:02:39.663093 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:02:39.663067    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5806796f_e98d_4eb1_b24c_d4c0cfc057f4.slice/crio-605cfa2f89d3ab39191e9507716386203ecebddfbe129f696ff146970c375629 WatchSource:0}: Error finding container 605cfa2f89d3ab39191e9507716386203ecebddfbe129f696ff146970c375629: Status 404 returned error can't find the container with id 605cfa2f89d3ab39191e9507716386203ecebddfbe129f696ff146970c375629
Nov 03 16:02:39.663691 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:02:39.663668    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61353437_1060_48e2_89f0_b51beb1fdb60.slice/crio-78a6603de7ff47bc15888ece6c1728a5f9cae64f61d8a7b06abb4c1785c1b56e WatchSource:0}: Error finding container 78a6603de7ff47bc15888ece6c1728a5f9cae64f61d8a7b06abb4c1785c1b56e: Status 404 returned error can't find the container with id 78a6603de7ff47bc15888ece6c1728a5f9cae64f61d8a7b06abb4c1785c1b56e
Nov 03 16:02:39.738492 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:39.738360    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8vvpp"]
Nov 03 16:02:39.742499 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:02:39.742475    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f7db5a_c0f7_480e_a19f_8b77e7285af2.slice/crio-c9460f4260a6754bf5741b5addc46c13a81142ddf334c3010eae472814c46844 WatchSource:0}: Error finding container c9460f4260a6754bf5741b5addc46c13a81142ddf334c3010eae472814c46844: Status 404 returned error can't find the container with id c9460f4260a6754bf5741b5addc46c13a81142ddf334c3010eae472814c46844
Nov 03 16:02:40.002363 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.002273    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hdzxp" event={"ID":"61353437-1060-48e2-89f0-b51beb1fdb60","Type":"ContainerStarted","Data":"5132718e85adecf5a8acf13153b20b47fd64c48004d4102059c4cd073c61fe55"}
Nov 03 16:02:40.002363 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.002358    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hdzxp" event={"ID":"61353437-1060-48e2-89f0-b51beb1fdb60","Type":"ContainerStarted","Data":"78a6603de7ff47bc15888ece6c1728a5f9cae64f61d8a7b06abb4c1785c1b56e"}
Nov 03 16:02:40.003419 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.003387    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8vvpp" event={"ID":"a9f7db5a-c0f7-480e-a19f-8b77e7285af2","Type":"ContainerStarted","Data":"c9460f4260a6754bf5741b5addc46c13a81142ddf334c3010eae472814c46844"}
Nov 03 16:02:40.004418 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.004398    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kgdhx" event={"ID":"5806796f-e98d-4eb1-b24c-d4c0cfc057f4","Type":"ContainerStarted","Data":"605cfa2f89d3ab39191e9507716386203ecebddfbe129f696ff146970c375629"}
Nov 03 16:02:40.259942 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.259627    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:40.655531 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.655472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:40.785471 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.785442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:02:40.785650 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.785515    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:02:40.785650 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.785535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:02:40.792943 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.792740    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 16:02:40.792943 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.792904    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 16:02:40.793139 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.792952    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c8h5g\""
Nov 03 16:02:40.797847 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.797819    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 16:02:40.798044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.797858    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 16:02:40.798044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.797858    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d4csl\""
Nov 03 16:02:40.798044 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.797911    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xkdws\""
Nov 03 16:02:40.808492 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.808467    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:40.974276 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:40.974240    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:41.009551 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:41.009508    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hdzxp" event={"ID":"61353437-1060-48e2-89f0-b51beb1fdb60","Type":"ContainerStarted","Data":"745b0fedabf00316e852cb2bc9727d117039721839169db3c67b62cd56e5b1b1"}
Nov 03 16:02:42.014676 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.014608    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kgdhx" event={"ID":"5806796f-e98d-4eb1-b24c-d4c0cfc057f4","Type":"ContainerStarted","Data":"15f2f30595a8b07ca31e4a1a9dee3ebfabed97a6137220b46b1316c42bb290d8"}
Nov 03 16:02:42.014676 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.014660    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kgdhx" event={"ID":"5806796f-e98d-4eb1-b24c-d4c0cfc057f4","Type":"ContainerStarted","Data":"d0bf8f1f7d3e9756f3fd6b1df05f0582eba2729776ec728d0edb7fc7153f95f3"}
Nov 03 16:02:42.015245 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.014768    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:42.040642 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.040603    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-kgdhx" podStartSLOduration=1.102325498 podStartE2EDuration="3.040591566s" totalImagesPullingTime="1.938266068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:42.040291596 +0000 UTC m=+47.866521025" watchObservedRunningTime="2026-11-03 16:02:42.040591566 +0000 UTC m=+47.866820993"
Nov 03 16:02:42.434122 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.434090    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:42.621256 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.621225    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lvq8q"]
Nov 03 16:02:42.623329 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.623311    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.631444 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.631419    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"]
Nov 03 16:02:42.633162 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.633143    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.647680 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.647655    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"]
Nov 03 16:02:42.672123 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.672084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:42.677636 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.677614    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xrtx4\""
Nov 03 16:02:42.677777 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.677635    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 16:02:42.677777 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.677642    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 16:02:42.677777 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.677656    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 16:02:42.677777 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.677669    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 16:02:42.677777 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.677685    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-56b2k\""
Nov 03 16:02:42.723934 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.723787    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 16:02:42.723934 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.723865    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 16:02:42.724065 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.723985    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 16:02:42.724065 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.724040    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 16:02:42.724139 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.724071    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 16:02:42.738316 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.738288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.738394 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.738319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.738394 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.738348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-sys\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.738491 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.738476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-wtmp\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.738530 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.738500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9c83af76-9520-4c3d-8aec-3590c97abaf7-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.738530 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.738519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.738599 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.738537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5r8x\" (UniqueName: \"kubernetes.io/projected/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-api-access-q5r8x\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.738599 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.738587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f0811b4-9b91-454b-b715-6aaf519b5e92-metrics-client-ca\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.738670 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.738620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.738670 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.738649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c83af76-9520-4c3d-8aec-3590c97abaf7-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.738740 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.738699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-748gh\" (UniqueName: \"kubernetes.io/projected/9f0811b4-9b91-454b-b715-6aaf519b5e92-kube-api-access-748gh\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.738740 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.738715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.738817 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.738764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-textfile\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.738817 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.738788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-root\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.738892 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.738829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-tls\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.839469 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.839435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-748gh\" (UniqueName: \"kubernetes.io/projected/9f0811b4-9b91-454b-b715-6aaf519b5e92-kube-api-access-748gh\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.839621 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.839472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.839621 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.839503    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-textfile\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.839730 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.839680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-root\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.839780 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.839715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-tls\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.839780 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.839746    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-root\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.839914 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.839790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.839914 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.839840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.839914 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.839872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-sys\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.839914 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.839904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-wtmp\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.840105 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.839867    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-textfile\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.840105 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.839924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9c83af76-9520-4c3d-8aec-3590c97abaf7-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.840105 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.839983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.840105 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.840016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5r8x\" (UniqueName: \"kubernetes.io/projected/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-api-access-q5r8x\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.840105 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.840072    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-wtmp\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.840105 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.840083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f0811b4-9b91-454b-b715-6aaf519b5e92-metrics-client-ca\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.840379 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.840111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.840379 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.840226    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9c83af76-9520-4c3d-8aec-3590c97abaf7-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.840379 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.840281    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-sys\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.840654 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.840586    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.840654 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.840613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c83af76-9520-4c3d-8aec-3590c97abaf7-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.840828 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.840615    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.840945 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.840920    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f0811b4-9b91-454b-b715-6aaf519b5e92-metrics-client-ca\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.841218 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.841166    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c83af76-9520-4c3d-8aec-3590c97abaf7-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.842969 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.842952    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.843090 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.843064    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.843397 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.843378    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-tls\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.843453 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.843439    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.846051 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.845993    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:42.854381 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.854362    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-748gh\" (UniqueName: \"kubernetes.io/projected/9f0811b4-9b91-454b-b715-6aaf519b5e92-kube-api-access-748gh\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.856372 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.856349    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5r8x\" (UniqueName: \"kubernetes.io/projected/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-api-access-q5r8x\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:42.937118 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.937084    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:02:42.946173 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:02:42.946146    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0811b4_9b91_454b_b715_6aaf519b5e92.slice/crio-b9c0ea4d51039c6b39d99d735c2078885371f2ab3771a3b92a44db93c6c9cff4 WatchSource:0}: Error finding container b9c0ea4d51039c6b39d99d735c2078885371f2ab3771a3b92a44db93c6c9cff4: Status 404 returned error can't find the container with id b9c0ea4d51039c6b39d99d735c2078885371f2ab3771a3b92a44db93c6c9cff4
Nov 03 16:02:42.946769 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:42.946751    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:02:43.005723 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:43.005691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:43.018968 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:43.018934    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvq8q" event={"ID":"9f0811b4-9b91-454b-b715-6aaf519b5e92","Type":"ContainerStarted","Data":"b9c0ea4d51039c6b39d99d735c2078885371f2ab3771a3b92a44db93c6c9cff4"}
Nov 03 16:02:43.020446 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:43.020423    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8vvpp" event={"ID":"a9f7db5a-c0f7-480e-a19f-8b77e7285af2","Type":"ContainerStarted","Data":"a9eef9a8fdff2e58e591117997e85a9674cf1b9da185d6d09d07784db0ee72ec"}
Nov 03 16:02:43.022370 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:43.022350    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hdzxp" event={"ID":"61353437-1060-48e2-89f0-b51beb1fdb60","Type":"ContainerStarted","Data":"92cf77b03944763e0138d999d964e7e30f4857a5e81af9b77470c1ae8dc66dc7"}
Nov 03 16:02:43.040672 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:43.040631    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8vvpp" podStartSLOduration=1.335788962 podStartE2EDuration="4.040619196s" totalImagesPullingTime="2.704830234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:43.039843231 +0000 UTC m=+48.866072658" watchObservedRunningTime="2026-11-03 16:02:43.040619196 +0000 UTC m=+48.866848622"
Nov 03 16:02:43.061220 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:43.061180    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hdzxp" podStartSLOduration=1.855410124 podStartE2EDuration="4.061167553s" totalImagesPullingTime="2.205757429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:02:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:43.060711412 +0000 UTC m=+48.886940842" watchObservedRunningTime="2026-11-03 16:02:43.061167553 +0000 UTC m=+48.887396979"
Nov 03 16:02:43.159280 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:43.159243    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:43.220120 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:43.220093    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"]
Nov 03 16:02:43.222545 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:02:43.222521    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c83af76_9520_4c3d_8aec_3590c97abaf7.slice/crio-cd13036ac6e4a5fa595a87799d6e14ccb8b2b8e2c23890c9d58ed3889eb7ed26 WatchSource:0}: Error finding container cd13036ac6e4a5fa595a87799d6e14ccb8b2b8e2c23890c9d58ed3889eb7ed26: Status 404 returned error can't find the container with id cd13036ac6e4a5fa595a87799d6e14ccb8b2b8e2c23890c9d58ed3889eb7ed26
Nov 03 16:02:43.307754 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:43.307718    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:43.676619 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:43.676534    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:43.891031 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:43.891003    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:44.026681 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.026601    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq" event={"ID":"9c83af76-9520-4c3d-8aec-3590c97abaf7","Type":"ContainerStarted","Data":"cd13036ac6e4a5fa595a87799d6e14ccb8b2b8e2c23890c9d58ed3889eb7ed26"}
Nov 03 16:02:44.028949 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.028925    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvq8q" event={"ID":"9f0811b4-9b91-454b-b715-6aaf519b5e92","Type":"ContainerStarted","Data":"61e8aa3d32e461ba292b5f720298e82d4164016bdf87189fb4b6e08d39bbd84a"}
Nov 03 16:02:44.739680 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.739641    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7747889c6f-6psbm"]
Nov 03 16:02:44.743539 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.743514    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.753358 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.753335    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b791nd51864r5\""
Nov 03 16:02:44.753447 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.753416    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 16:02:44.753523 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.753511    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 16:02:44.753618 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.753605    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 16:02:44.753675 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.753615    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-d2hch\""
Nov 03 16:02:44.753720 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.753691    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 16:02:44.753720 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.753697    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 16:02:44.767776 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.763947    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7747889c6f-6psbm"]
Nov 03 16:02:44.857333 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.857263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-tls\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.857333 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.857296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-grpc-tls\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.857333 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.857318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc926104-1c6c-46f3-9d8a-7f1d8394d713-tmp\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.857559 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.857395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.857559 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.857508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.857658 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.857587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.857658 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.857634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc926104-1c6c-46f3-9d8a-7f1d8394d713-metrics-client-ca\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.857750 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.857664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfqzs\" (UniqueName: \"kubernetes.io/projected/cc926104-1c6c-46f3-9d8a-7f1d8394d713-kube-api-access-pfqzs\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.857750 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.857698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.958587 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.958542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-grpc-tls\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.958587 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.958588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc926104-1c6c-46f3-9d8a-7f1d8394d713-tmp\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.958774 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.958719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.958847 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.958825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.958880 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.958867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.958923 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.958892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc926104-1c6c-46f3-9d8a-7f1d8394d713-metrics-client-ca\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.958923 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.958915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfqzs\" (UniqueName: \"kubernetes.io/projected/cc926104-1c6c-46f3-9d8a-7f1d8394d713-kube-api-access-pfqzs\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.959020 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.958944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.959020 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.958992    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc926104-1c6c-46f3-9d8a-7f1d8394d713-tmp\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.959020 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.959002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-tls\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.959751 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.959712    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc926104-1c6c-46f3-9d8a-7f1d8394d713-metrics-client-ca\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.961603 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.961556    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.961966 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.961944    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.962046 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.962014    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.962190 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.962153    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-tls\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.962248 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.962200    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-grpc-tls\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.962457 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.962441    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:44.972026 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:44.971988    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfqzs\" (UniqueName: \"kubernetes.io/projected/cc926104-1c6c-46f3-9d8a-7f1d8394d713-kube-api-access-pfqzs\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:45.037213 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:45.037171    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq" event={"ID":"9c83af76-9520-4c3d-8aec-3590c97abaf7","Type":"ContainerStarted","Data":"03a2d30b4bede0e617b521b5b0e18af33d41fa800aa931a808664bd4cc43f75b"}
Nov 03 16:02:45.037656 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:45.037243    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq" event={"ID":"9c83af76-9520-4c3d-8aec-3590c97abaf7","Type":"ContainerStarted","Data":"5d64e413d074b39fabb4a3ddf898c0a5bd44cdf4e6286c80b401a6e6679ba36f"}
Nov 03 16:02:45.037656 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:45.037269    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq" event={"ID":"9c83af76-9520-4c3d-8aec-3590c97abaf7","Type":"ContainerStarted","Data":"c75c01fefd89b101d2d60b80510411781737555d46d75a90de3442f38f05e21f"}
Nov 03 16:02:45.039327 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:45.039302    2577 generic.go:356] "Generic (PLEG): container finished" podID="9f0811b4-9b91-454b-b715-6aaf519b5e92" containerID="61e8aa3d32e461ba292b5f720298e82d4164016bdf87189fb4b6e08d39bbd84a" exitCode=0
Nov 03 16:02:45.039459 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:45.039355    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvq8q" event={"ID":"9f0811b4-9b91-454b-b715-6aaf519b5e92","Type":"ContainerDied","Data":"61e8aa3d32e461ba292b5f720298e82d4164016bdf87189fb4b6e08d39bbd84a"}
Nov 03 16:02:45.039459 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:45.039386    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvq8q" event={"ID":"9f0811b4-9b91-454b-b715-6aaf519b5e92","Type":"ContainerStarted","Data":"9b4979657ed7552be34f550460b004146900f729b244cf9b4eabd20221c66ffb"}
Nov 03 16:02:45.039459 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:45.039401    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvq8q" event={"ID":"9f0811b4-9b91-454b-b715-6aaf519b5e92","Type":"ContainerStarted","Data":"5f4b7187877858cb27bcc9610ec69021b9c95210db54828384696b788b4d429e"}
Nov 03 16:02:45.071828 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:45.071770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:45.073180 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:45.073089    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq" podStartSLOduration=1.738583695 podStartE2EDuration="3.073077181s" totalImagesPullingTime="1.334493486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:45.069216346 +0000 UTC m=+50.895445774" watchObservedRunningTime="2026-11-03 16:02:45.073077181 +0000 UTC m=+50.899306609"
Nov 03 16:02:45.101551 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:45.101506    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lvq8q" podStartSLOduration=2.101574334 podStartE2EDuration="3.101494163s" totalImagesPullingTime="916.641172ms" totalInitContainerRuntime="83.278657ms" isStatefulPod=true podCreationTimestamp="2026-11-03 16:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:45.100856182 +0000 UTC m=+50.927085606" watchObservedRunningTime="2026-11-03 16:02:45.101494163 +0000 UTC m=+50.927723618"
Nov 03 16:02:45.179593 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:45.179563    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:45.347014 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:45.346974    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:45.355939 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:45.355919    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7747889c6f-6psbm"]
Nov 03 16:02:45.358066 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:02:45.358041    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc926104_1c6c_46f3_9d8a_7f1d8394d713.slice/crio-46959e3250cc1eaada8d5b775cc0246c475b65a4363584dc4836bb5ac1dd8534 WatchSource:0}: Error finding container 46959e3250cc1eaada8d5b775cc0246c475b65a4363584dc4836bb5ac1dd8534: Status 404 returned error can't find the container with id 46959e3250cc1eaada8d5b775cc0246c475b65a4363584dc4836bb5ac1dd8534
Nov 03 16:02:45.689690 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:45.689655    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:45.918032 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:45.918000    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:46.044367 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:46.044332    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm" event={"ID":"cc926104-1c6c-46f3-9d8a-7f1d8394d713","Type":"ContainerStarted","Data":"46959e3250cc1eaada8d5b775cc0246c475b65a4363584dc4836bb5ac1dd8534"}
Nov 03 16:02:46.075317 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:46.075288    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:47.120095 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.120058    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"]
Nov 03 16:02:47.123434 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.123407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.128047 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.128026    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 16:02:47.128047 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.128026    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 16:02:47.159620 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.159590    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5jf2w\""
Nov 03 16:02:47.159620 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.159615    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 16:02:47.159834 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.159659    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 16:02:47.159834 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.159714    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ei2tuiaibn95i\""
Nov 03 16:02:47.160077 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.160014    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"]
Nov 03 16:02:47.279520 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.279485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw9h4\" (UniqueName: \"kubernetes.io/projected/1e20bd59-e885-4662-bb6b-7cd403b9d752-kube-api-access-fw9h4\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.279751 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.279528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-secret-metrics-server-client-certs\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.279751 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.279589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-secret-metrics-server-tls\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.279751 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.279637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-client-ca-bundle\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.279751 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.279675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1e20bd59-e885-4662-bb6b-7cd403b9d752-audit-log\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.279751 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.279706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1e20bd59-e885-4662-bb6b-7cd403b9d752-metrics-server-audit-profiles\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.279751 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.279728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e20bd59-e885-4662-bb6b-7cd403b9d752-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.380731 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.380647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw9h4\" (UniqueName: \"kubernetes.io/projected/1e20bd59-e885-4662-bb6b-7cd403b9d752-kube-api-access-fw9h4\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.380731 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.380688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-secret-metrics-server-client-certs\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.380731 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.380713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-secret-metrics-server-tls\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.380998 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.380743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-client-ca-bundle\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.380998 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.380783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1e20bd59-e885-4662-bb6b-7cd403b9d752-audit-log\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.380998 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.380935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1e20bd59-e885-4662-bb6b-7cd403b9d752-metrics-server-audit-profiles\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.381153 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.381020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e20bd59-e885-4662-bb6b-7cd403b9d752-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.381244 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.381227    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1e20bd59-e885-4662-bb6b-7cd403b9d752-audit-log\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.381860 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.381836    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e20bd59-e885-4662-bb6b-7cd403b9d752-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.381962 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.381924    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1e20bd59-e885-4662-bb6b-7cd403b9d752-metrics-server-audit-profiles\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.384146 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.384127    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-client-ca-bundle\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.384353 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.384306    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-secret-metrics-server-client-certs\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.384543 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.384521    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-secret-metrics-server-tls\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.392863 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.392842    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw9h4\" (UniqueName: \"kubernetes.io/projected/1e20bd59-e885-4662-bb6b-7cd403b9d752-kube-api-access-fw9h4\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.460259 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.460225    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:02:47.503273 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.503239    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:47.654074 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.653999    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:47.746122 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:47.745977    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"]
Nov 03 16:02:47.748063 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:02:47.748039    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e20bd59_e885_4662_bb6b_7cd403b9d752.slice/crio-dfb883769f66b407fc016c03dffa3a4dc0231923b0820cb721c0aae7d6e71ff6 WatchSource:0}: Error finding container dfb883769f66b407fc016c03dffa3a4dc0231923b0820cb721c0aae7d6e71ff6: Status 404 returned error can't find the container with id dfb883769f66b407fc016c03dffa3a4dc0231923b0820cb721c0aae7d6e71ff6
Nov 03 16:02:48.056431 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.056400    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm" event={"ID":"cc926104-1c6c-46f3-9d8a-7f1d8394d713","Type":"ContainerStarted","Data":"e4a0fb50363a092c6cf26f63be370cf663ae01f9978fb03be2d274f543d9d12a"}
Nov 03 16:02:48.056581 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.056492    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm" event={"ID":"cc926104-1c6c-46f3-9d8a-7f1d8394d713","Type":"ContainerStarted","Data":"86f78d407bacc3d2a956f6b2f6efde9cc14813ced1aaada6fb23f17967326aaf"}
Nov 03 16:02:48.056581 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.056525    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm" event={"ID":"cc926104-1c6c-46f3-9d8a-7f1d8394d713","Type":"ContainerStarted","Data":"7e864af45427d475a20f11ac0924d39860a8a6922b25acd8853e8d2d55a90e21"}
Nov 03 16:02:48.057537 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.057513    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch" event={"ID":"1e20bd59-e885-4662-bb6b-7cd403b9d752","Type":"ContainerStarted","Data":"dfb883769f66b407fc016c03dffa3a4dc0231923b0820cb721c0aae7d6e71ff6"}
Nov 03 16:02:48.962033 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.961999    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 16:02:48.965808 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.965773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.971842 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.971815    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 16:02:48.974364 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.974062    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 16:02:48.974364 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.974087    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8qdq8\""
Nov 03 16:02:48.974364 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.974142    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 16:02:48.974364 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.974155    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 16:02:48.974364 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.974189    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 16:02:48.974364 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.974225    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 16:02:48.974364 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.974234    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-duq0m103irmgu\""
Nov 03 16:02:48.974364 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.974281    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 16:02:48.974364 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.974289    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 16:02:48.974364 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.974149    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 16:02:48.974364 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.974308    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 16:02:48.974876 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.974448    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 16:02:48.974876 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.974475    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 16:02:48.985307 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.985286    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 16:02:48.985459 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.985442    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 16:02:48.988746 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.988723    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 16:02:48.993490 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.993463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.993614 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.993510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.993614 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.993540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.993718 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.993636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config-out\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.993718 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.993653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.993718 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.993677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.993879 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.993720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfqjb\" (UniqueName: \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-kube-api-access-kfqjb\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.993879 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.993856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.993974 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.993881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.993974 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.993940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.994066 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.993983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-web-config\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.994066 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.994012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.994066 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.994039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.994204 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.994079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.994204 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.994108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.994204 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.994154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.994204 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.994200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.994400 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.994219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.994400 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.994235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:48.994400 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:48.994280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.027939 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.027907    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:49.064645 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.064618    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm" event={"ID":"cc926104-1c6c-46f3-9d8a-7f1d8394d713","Type":"ContainerStarted","Data":"7d838467fac0fff7de8c7cc05d91dd1303b24dd0869d5ac1d93f0bfb9cccff65"}
Nov 03 16:02:49.064759 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.064689    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm" event={"ID":"cc926104-1c6c-46f3-9d8a-7f1d8394d713","Type":"ContainerStarted","Data":"4f80fda416a9e647d2f13494f03e4e81ba3e9d2b56bf284dfad4d625a333c6e8"}
Nov 03 16:02:49.064759 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.064730    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm" event={"ID":"cc926104-1c6c-46f3-9d8a-7f1d8394d713","Type":"ContainerStarted","Data":"06d5fb07640707065391bf58bd9e01d2e8f0bcb7682f3fe19ffdfecb48cbb42d"}
Nov 03 16:02:49.064884 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.064823    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:49.094541 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.094499    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm" podStartSLOduration=2.142858812 podStartE2EDuration="5.094487973s" totalImagesPullingTime="2.951629161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:49.092757607 +0000 UTC m=+54.918987033" watchObservedRunningTime="2026-11-03 16:02:49.094487973 +0000 UTC m=+54.920717449"
Nov 03 16:02:49.094962 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.094943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config-out\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095024 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.094972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095102 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095132 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfqjb\" (UniqueName: \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-kube-api-access-kfqjb\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095193 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095323 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095433 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095513 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-web-config\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095513 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095621 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095621 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095621 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095755 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095755 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095922 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095922 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095922 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.095922 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.096101 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.096101 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.095999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.096616 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.096588    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.097246 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.097220    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.098406 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.098382    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.100200 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.100179    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.100564 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.100544    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.100853 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.100822    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.101075 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.101053    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.103306 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.103286    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.103486 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.103465    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config-out\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.103688 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.103669    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.103988 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.103971    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.105701 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.105681    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.106562 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.106473    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.106562 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.106493    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.106562 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.106541    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-web-config\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.106904 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.106844    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.106989 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.106937    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.108220 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.108191    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.108349 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.108328    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.109252 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.109232    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfqjb\" (UniqueName: \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-kube-api-access-kfqjb\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.292288 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.292252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:49.587766 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.587387    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 16:02:49.590670 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:02:49.590635    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587c9773_ff2c_4c1d_a4ed_f51fc12ee704.slice/crio-3c1e5ce477e74f85116c5361dffe10a499ce6966005b6620de86c37ed3734ae9 WatchSource:0}: Error finding container 3c1e5ce477e74f85116c5361dffe10a499ce6966005b6620de86c37ed3734ae9: Status 404 returned error can't find the container with id 3c1e5ce477e74f85116c5361dffe10a499ce6966005b6620de86c37ed3734ae9
Nov 03 16:02:49.973292 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:49.973221    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:02:50.069859 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:50.069822    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch" event={"ID":"1e20bd59-e885-4662-bb6b-7cd403b9d752","Type":"ContainerStarted","Data":"b16b39f38cb7d9a05cb2f7602da61c97eb57606e26b1076e50f8c08f054d3199"}
Nov 03 16:02:50.071206 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:50.071171    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"587c9773-ff2c-4c1d-a4ed-f51fc12ee704","Type":"ContainerStarted","Data":"3c1e5ce477e74f85116c5361dffe10a499ce6966005b6620de86c37ed3734ae9"}
Nov 03 16:02:50.204479 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:50.204435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:50.383944 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:50.383896    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:50.549587 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:50.549548    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:50.767485 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:50.767444    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:51.076025 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:51.075993    2577 generic.go:356] "Generic (PLEG): container finished" podID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerID="916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9" exitCode=0
Nov 03 16:02:51.076376 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:51.076081    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"587c9773-ff2c-4c1d-a4ed-f51fc12ee704","Type":"ContainerDied","Data":"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9"}
Nov 03 16:02:51.112287 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:51.112244    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch" podStartSLOduration=2.4305263 podStartE2EDuration="4.112225747s" totalImagesPullingTime="1.681699447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:50.099849869 +0000 UTC m=+55.926079296" watchObservedRunningTime="2026-11-03 16:02:51.112225747 +0000 UTC m=+56.938455175"
Nov 03 16:02:52.024546 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:52.024516    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:02:53.265643 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:53.265605    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:53.422364 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:53.422325    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:53.898480 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:53.898427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:54.063993 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:54.063944    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:54.903944 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:54.903908    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:55.059672 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:55.059641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:55.076913 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:55.076888    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:02:55.093169 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:55.093139    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"587c9773-ff2c-4c1d-a4ed-f51fc12ee704","Type":"ContainerStarted","Data":"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"}
Nov 03 16:02:55.093269 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:55.093194    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"587c9773-ff2c-4c1d-a4ed-f51fc12ee704","Type":"ContainerStarted","Data":"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"}
Nov 03 16:02:55.093269 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:55.093221    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"587c9773-ff2c-4c1d-a4ed-f51fc12ee704","Type":"ContainerStarted","Data":"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"}
Nov 03 16:02:55.093269 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:55.093255    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"587c9773-ff2c-4c1d-a4ed-f51fc12ee704","Type":"ContainerStarted","Data":"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"}
Nov 03 16:02:55.093362 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:55.093278    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"587c9773-ff2c-4c1d-a4ed-f51fc12ee704","Type":"ContainerStarted","Data":"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"}
Nov 03 16:02:55.093362 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:55.093301    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"587c9773-ff2c-4c1d-a4ed-f51fc12ee704","Type":"ContainerStarted","Data":"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"}
Nov 03 16:02:55.147207 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:55.147144    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.142260539 podStartE2EDuration="7.147129375s" totalImagesPullingTime="3.987977869s" totalInitContainerRuntime="16.890967ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:55.146182526 +0000 UTC m=+60.972411956" watchObservedRunningTime="2026-11-03 16:02:55.147129375 +0000 UTC m=+60.973358803"
Nov 03 16:02:56.154617 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:56.154566    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:56.311046 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:56.311006    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:57.282278 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:57.282246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:57.446713 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:57.446678    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:59.121262 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:59.121221    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:59.280067 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:59.280019    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:59.292377 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:59.292346    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:02:59.822117 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:59.822085    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:02:59.977772 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:02:59.977742    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:00.402888 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.402855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:03:00.408275 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.408253    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 16:03:00.416332 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.416309    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:03:00.503243 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.503203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4774\" (UniqueName: \"kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774\") pod \"network-check-target-2ht7q\" (UID: \"af63014e-5ed6-4213-8838-529749c931cb\") " pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:03:00.503394 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.503290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:03:00.506298 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.506278    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 16:03:00.506444 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.506431    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 16:03:00.516490 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.516466    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:03:00.516647 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.516632    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 16:03:00.526925 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.526906    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4774\" (UniqueName: \"kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774\") pod \"network-check-target-2ht7q\" (UID: \"af63014e-5ed6-4213-8838-529749c931cb\") " pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:03:00.605705 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.605681    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d4csl\""
Nov 03 16:03:00.612876 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.612854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:03:00.614105 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.614088    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c8h5g\""
Nov 03 16:03:00.621263 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.621236    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:03:00.621496 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.621480    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xkdws\""
Nov 03 16:03:00.629152 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.629119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:03:00.901390 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.901361    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2ht7q"]
Nov 03 16:03:00.904052 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.904022    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qmd98"]
Nov 03 16:03:00.904629 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:03:00.904584    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf63014e_5ed6_4213_8838_529749c931cb.slice/crio-afdc0a4848ba4fbb0befd56b1d5712ec0b6d92fe8703fbd426a5962594b34a04 WatchSource:0}: Error finding container afdc0a4848ba4fbb0befd56b1d5712ec0b6d92fe8703fbd426a5962594b34a04: Status 404 returned error can't find the container with id afdc0a4848ba4fbb0befd56b1d5712ec0b6d92fe8703fbd426a5962594b34a04
Nov 03 16:03:00.907156 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:03:00.907134    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea5065d_e1c6_4152_842c_833cd8155772.slice/crio-bf3d135781ee7be6c7d6f89592858a5911d2ea65f7ab1379bab219ec9134cb74 WatchSource:0}: Error finding container bf3d135781ee7be6c7d6f89592858a5911d2ea65f7ab1379bab219ec9134cb74: Status 404 returned error can't find the container with id bf3d135781ee7be6c7d6f89592858a5911d2ea65f7ab1379bab219ec9134cb74
Nov 03 16:03:00.927406 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:00.927347    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sf4cc"]
Nov 03 16:03:00.929774 ip-10-0-137-159 kubenswrapper[2577]: W1103 16:03:00.929751    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00bade6_34ba_4d87_a740_9727b8372650.slice/crio-f75f297c8ad68d2c4a0c864db203eb147ee523fd6577e564d6462907f853962a WatchSource:0}: Error finding container f75f297c8ad68d2c4a0c864db203eb147ee523fd6577e564d6462907f853962a: Status 404 returned error can't find the container with id f75f297c8ad68d2c4a0c864db203eb147ee523fd6577e564d6462907f853962a
Nov 03 16:03:01.114837 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:01.114779    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sf4cc" event={"ID":"a00bade6-34ba-4d87-a740-9727b8372650","Type":"ContainerStarted","Data":"f75f297c8ad68d2c4a0c864db203eb147ee523fd6577e564d6462907f853962a"}
Nov 03 16:03:01.115896 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:01.115868    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2ht7q" event={"ID":"af63014e-5ed6-4213-8838-529749c931cb","Type":"ContainerStarted","Data":"afdc0a4848ba4fbb0befd56b1d5712ec0b6d92fe8703fbd426a5962594b34a04"}
Nov 03 16:03:01.116963 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:01.116935    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qmd98" event={"ID":"7ea5065d-e1c6-4152-842c-833cd8155772","Type":"ContainerStarted","Data":"bf3d135781ee7be6c7d6f89592858a5911d2ea65f7ab1379bab219ec9134cb74"}
Nov 03 16:03:01.276167 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:01.276140    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:01.428868 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:01.428835    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:03.133571 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:03.133530    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sf4cc" event={"ID":"a00bade6-34ba-4d87-a740-9727b8372650","Type":"ContainerStarted","Data":"da8392fd3ea4521832067fce74ee9186ef99dad21310d9222722a65c71b4a118"}
Nov 03 16:03:03.134077 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:03.133638    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sf4cc" event={"ID":"a00bade6-34ba-4d87-a740-9727b8372650","Type":"ContainerStarted","Data":"9c7e68739da988e0304cc120d76ae04a3358e951a5baabfd5b78cdfc2360e6ad"}
Nov 03 16:03:03.155520 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:03.155457    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sf4cc" podStartSLOduration=67.968407177 podStartE2EDuration="1m9.155438443s" totalImagesPullingTime="1.187031266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:03:03.153003095 +0000 UTC m=+68.979232523" watchObservedRunningTime="2026-11-03 16:03:03.155438443 +0000 UTC m=+68.981667872"
Nov 03 16:03:03.338272 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:03.338232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:03.494169 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:03.493995    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:05.143900 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:05.143859    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2ht7q" event={"ID":"af63014e-5ed6-4213-8838-529749c931cb","Type":"ContainerStarted","Data":"7d415eade44a8c33d8f21057fbc1649f675aa12bfad984d508fe9e333637cf54"}
Nov 03 16:03:05.144376 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:05.143998    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:03:05.164576 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:05.164525    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2ht7q" podStartSLOduration=67.853885023 podStartE2EDuration="1m11.164509934s" totalImagesPullingTime="3.310624911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:03:05.163749201 +0000 UTC m=+70.989978630" watchObservedRunningTime="2026-11-03 16:03:05.164509934 +0000 UTC m=+70.990739363"
Nov 03 16:03:05.924850 ip-10-0-137-159 kubenswrapper[2577]: I1103 16:03:05.924749    2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 16:03:05.924922 ip-10-0-137-159 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 16:03:05.931493 ip-10-0-137-159 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 16:03:05.931662 ip-10-0-137-159 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 16:03:05.931858 ip-10-0-137-159 systemd[1]: kubelet.service: Consumed 4.381s CPU time.
Nov 03 16:03:05.944485 ip-10-0-137-159 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 16:03:06.015559 ip-10-0-137-159 kubenswrapper[6506]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:03:06.015559 ip-10-0-137-159 kubenswrapper[6506]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 16:03:06.015559 ip-10-0-137-159 kubenswrapper[6506]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:03:06.015559 ip-10-0-137-159 kubenswrapper[6506]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:03:06.017958 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.017877    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:03:06.017958 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.017955    6506 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:03:06.017958 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.017960    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:03:06.017958 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.017965    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.017969    6506 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.017973    6506 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.017977    6506 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.017981    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.017985    6506 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.017988    6506 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.017992    6506 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.017996    6506 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018000    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018004    6506 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018008    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018012    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018016    6506 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018019    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018024    6506 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018028    6506 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018032    6506 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018035    6506 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:03:06.018225 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018040    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018044    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018048    6506 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018052    6506 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018056    6506 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018065    6506 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018069    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018073    6506 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018080    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018088    6506 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018094    6506 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018099    6506 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018104    6506 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018108    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018113    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018117    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018121    6506 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018125    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:03:06.019077 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018132    6506 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018137    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018143    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018148    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018152    6506 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018157    6506 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018161    6506 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018165    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018169    6506 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018173    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018177    6506 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018182    6506 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018188    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018193    6506 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018198    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018202    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018206    6506 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018211    6506 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018215    6506 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018219    6506 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:03:06.019953 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018223    6506 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018227    6506 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018232    6506 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018236    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018240    6506 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018244    6506 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018249    6506 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018254    6506 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018258    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018262    6506 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018267    6506 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018271    6506 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018275    6506 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018279    6506 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018285    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018289    6506 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018294    6506 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018299    6506 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018303    6506 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018307    6506 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:03:06.020849 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018312    6506 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018316    6506 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018320    6506 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018324    6506 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018328    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018336    6506 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018342    6506 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018346    6506 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018351    6506 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018355    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018359    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018363    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018368    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018372    6506 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018377    6506 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018382    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018387    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018391    6506 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018395    6506 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:03:06.021766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018400    6506 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018404    6506 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018408    6506 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018412    6506 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018416    6506 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018420    6506 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.018425    6506 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019673    6506 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019687    6506 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019692    6506 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019697    6506 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019702    6506 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019707    6506 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019711    6506 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019715    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019719    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019724    6506 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019728    6506 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019732    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:03:06.022709 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019744    6506 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019748    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019753    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019757    6506 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019762    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019767    6506 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019771    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019775    6506 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019780    6506 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019784    6506 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019788    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019813    6506 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019818    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019822    6506 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019826    6506 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019831    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019835    6506 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019839    6506 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019844    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:03:06.023602 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019848    6506 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019852    6506 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019856    6506 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019860    6506 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019866    6506 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019872    6506 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019876    6506 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019881    6506 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019885    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019890    6506 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019894    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019898    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019902    6506 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019906    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019916    6506 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019921    6506 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019925    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019932    6506 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019936    6506 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:03:06.024438 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019940    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019944    6506 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019948    6506 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019952    6506 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019956    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019960    6506 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019965    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019969    6506 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019973    6506 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019977    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019982    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019986    6506 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019990    6506 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019995    6506 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.019999    6506 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020003    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020007    6506 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020011    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020018    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020022    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:03:06.024954 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020026    6506 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020030    6506 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020034    6506 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020039    6506 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020043    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020047    6506 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020051    6506 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020055    6506 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020064    6506 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020069    6506 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020073    6506 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020078    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020082    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020086    6506 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020094    6506 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020098    6506 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020102    6506 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020106    6506 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020110    6506 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020114    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:03:06.025480 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020118    6506 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020122    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020127    6506 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020131    6506 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020135    6506 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020139    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020145    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020149    6506 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020153    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020165    6506 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020172    6506 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020176    6506 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020181    6506 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020185    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020189    6506 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.020193    6506 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021029    6506 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021038    6506 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021043    6506 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:03:06.026071 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021048    6506 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021053    6506 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021058    6506 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021063    6506 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021067    6506 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021072    6506 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021083    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021088    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021093    6506 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021097    6506 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021101    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021105    6506 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021109    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021113    6506 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021118    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021122    6506 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021126    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021130    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021135    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:03:06.026577 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021139    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021143    6506 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021147    6506 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021151    6506 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021159    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021163    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021167    6506 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021171    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021176    6506 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021180    6506 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021184    6506 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021188    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021193    6506 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021197    6506 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021202    6506 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021207    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021211    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021215    6506 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021219    6506 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021223    6506 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:03:06.027421 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021228    6506 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021232    6506 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021236    6506 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021240    6506 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021244    6506 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021248    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021252    6506 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021256    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021261    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021265    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021269    6506 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021273    6506 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021277    6506 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021283    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021287    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021292    6506 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021298    6506 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021303    6506 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021307    6506 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021310    6506 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:03:06.028027 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021315    6506 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021319    6506 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021323    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021327    6506 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021332    6506 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021336    6506 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021341    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021345    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021349    6506 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021354    6506 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021358    6506 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021362    6506 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021366    6506 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021370    6506 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021374    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021378    6506 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021383    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021387    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021391    6506 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:03:06.028517 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021395    6506 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021399    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021403    6506 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021407    6506 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021411    6506 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021415    6506 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021420    6506 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021423    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021428    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021434    6506 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021438    6506 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021442    6506 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021446    6506 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021451    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021455    6506 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021459    6506 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021463    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021472    6506 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021479    6506 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:03:06.029098 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021485    6506 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021490    6506 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021494    6506 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021499    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021503    6506 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.021507    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021647    6506 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021662    6506 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021672    6506 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021679    6506 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021686    6506 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021694    6506 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021700    6506 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021712    6506 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021717    6506 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021722    6506 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021727    6506 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021733    6506 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021738    6506 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021742    6506 flags.go:64] FLAG: --cgroup-root=""
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021747    6506 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021752    6506 flags.go:64] FLAG: --client-ca-file=""
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021758    6506 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 16:03:06.029638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021763    6506 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021768    6506 flags.go:64] FLAG: --cluster-domain=""
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021773    6506 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021777    6506 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021782    6506 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021788    6506 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021838    6506 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021843    6506 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021848    6506 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021854    6506 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021858    6506 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021863    6506 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021868    6506 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021875    6506 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021880    6506 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021888    6506 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021893    6506 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021898    6506 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021902    6506 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021907    6506 flags.go:64] FLAG: --enable-server="true"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021915    6506 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021922    6506 flags.go:64] FLAG: --event-burst="100"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021927    6506 flags.go:64] FLAG: --event-qps="50"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021932    6506 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 16:03:06.030331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021937    6506 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021942    6506 flags.go:64] FLAG: --eviction-hard=""
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021949    6506 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021953    6506 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021958    6506 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021963    6506 flags.go:64] FLAG: --eviction-soft=""
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021968    6506 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021974    6506 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021980    6506 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021985    6506 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021989    6506 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021994    6506 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.021998    6506 flags.go:64] FLAG: --feature-gates=""
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022005    6506 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022010    6506 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022015    6506 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022020    6506 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022025    6506 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022029    6506 flags.go:64] FLAG: --help="false"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022034    6506 flags.go:64] FLAG: --hostname-override="ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022039    6506 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022044    6506 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022053    6506 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 16:03:06.031060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022059    6506 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022065    6506 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022070    6506 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022075    6506 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022079    6506 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022087    6506 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022092    6506 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022097    6506 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022102    6506 flags.go:64] FLAG: --kube-reserved=""
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022107    6506 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022111    6506 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022116    6506 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022120    6506 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022125    6506 flags.go:64] FLAG: --lock-file=""
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022130    6506 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022134    6506 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022140    6506 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022150    6506 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022154    6506 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022159    6506 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022163    6506 flags.go:64] FLAG: --logging-format="text"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022168    6506 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022173    6506 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022178    6506 flags.go:64] FLAG: --manifest-url=""
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022182    6506 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 16:03:06.031702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022190    6506 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022196    6506 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022202    6506 flags.go:64] FLAG: --max-pods="110"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022208    6506 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022213    6506 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022217    6506 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022222    6506 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022227    6506 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022232    6506 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022237    6506 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022248    6506 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022253    6506 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022259    6506 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022264    6506 flags.go:64] FLAG: --pod-cidr=""
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022269    6506 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022273    6506 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022278    6506 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022282    6506 flags.go:64] FLAG: --port="10250"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022287    6506 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022292    6506 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0271e572ab51e3f93"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022297    6506 flags.go:64] FLAG: --qos-reserved=""
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022302    6506 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022307    6506 flags.go:64] FLAG: --register-node="true"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022313    6506 flags.go:64] FLAG: --register-with-taints=""
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022319    6506 flags.go:64] FLAG: --registry-burst="10"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022324    6506 flags.go:64] FLAG: --registry-qps="5"
Nov 03 16:03:06.032502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022329    6506 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022334    6506 flags.go:64] FLAG: --reserved-memory=""
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022340    6506 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022345    6506 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022349    6506 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022354    6506 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022358    6506 flags.go:64] FLAG: --runonce="false"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022363    6506 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022368    6506 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022373    6506 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022378    6506 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022383    6506 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022388    6506 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022392    6506 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022398    6506 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022403    6506 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022407    6506 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022412    6506 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022418    6506 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022423    6506 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022428    6506 flags.go:64] FLAG: --system-cgroups=""
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022432    6506 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022441    6506 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022445    6506 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022450    6506 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 16:03:06.033240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022456    6506 flags.go:64] FLAG: --tls-min-version=""
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022461    6506 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022466    6506 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022470    6506 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022477    6506 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022481    6506 flags.go:64] FLAG: --v="2"
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022488    6506 flags.go:64] FLAG: --version="false"
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022494    6506 flags.go:64] FLAG: --vmodule=""
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022501    6506 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.022506    6506 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022741    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022750    6506 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022755    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022759    6506 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022764    6506 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022768    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022772    6506 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022776    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022781    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022785    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022809    6506 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:03:06.034127 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022814    6506 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022819    6506 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022823    6506 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022827    6506 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022833    6506 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022838    6506 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022842    6506 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022846    6506 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022850    6506 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022854    6506 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022859    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022863    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022868    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022872    6506 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022876    6506 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022884    6506 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022890    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022897    6506 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022904    6506 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:03:06.034877 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022909    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022913    6506 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022918    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022923    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022927    6506 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022931    6506 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022936    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022940    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022945    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022949    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022953    6506 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022957    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022962    6506 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022966    6506 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022970    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022975    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022979    6506 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022985    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022990    6506 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:03:06.035373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022994    6506 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.022998    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023002    6506 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023006    6506 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023011    6506 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023015    6506 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023020    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023024    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023028    6506 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023034    6506 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023038    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023042    6506 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023047    6506 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023051    6506 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023055    6506 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023059    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023063    6506 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023068    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023072    6506 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023076    6506 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:03:06.035971 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023081    6506 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023085    6506 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023089    6506 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023094    6506 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023098    6506 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023102    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023106    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023110    6506 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023114    6506 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023119    6506 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023125    6506 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023130    6506 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023134    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023138    6506 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023142    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023147    6506 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023151    6506 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023155    6506 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023159    6506 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:03:06.036566 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023163    6506 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023168    6506 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023174    6506 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023179    6506 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023183    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023187    6506 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023191    6506 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023195    6506 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023200    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023204    6506 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023208    6506 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023213    6506 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023217    6506 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023221    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023225    6506 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023230    6506 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023234    6506 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:03:06.037307 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023238    6506 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.023278    6506 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023331    6506 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023337    6506 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023342    6506 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023346    6506 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023353    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023358    6506 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023362    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023366    6506 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023371    6506 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023375    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023379    6506 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023383    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023387    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023392    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:03:06.037978 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023396    6506 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023400    6506 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023405    6506 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023410    6506 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023414    6506 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023418    6506 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023422    6506 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023426    6506 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023430    6506 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023435    6506 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023439    6506 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023443    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023447    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023451    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023456    6506 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023460    6506 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023467    6506 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023473    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023480    6506 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:03:06.038407 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023487    6506 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023493    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023499    6506 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023505    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023511    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023515    6506 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023520    6506 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023524    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023528    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023533    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023537    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023541    6506 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023545    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023550    6506 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023554    6506 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023562    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023566    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023571    6506 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023575    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:03:06.038900 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023579    6506 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023583    6506 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023587    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023592    6506 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023596    6506 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023600    6506 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023604    6506 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023608    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023612    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023616    6506 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023620    6506 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023624    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023629    6506 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023634    6506 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023638    6506 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023642    6506 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023650    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023654    6506 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023659    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023663    6506 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:03:06.039499 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023667    6506 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023672    6506 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023676    6506 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023680    6506 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023685    6506 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023689    6506 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023693    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023698    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023702    6506 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023706    6506 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023711    6506 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023715    6506 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023719    6506 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023723    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023728    6506 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023732    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023736    6506 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023740    6506 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023744    6506 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:03:06.040082 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023748    6506 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023752    6506 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023756    6506 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023760    6506 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023764    6506 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023769    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023773    6506 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023777    6506 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023781    6506 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023788    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023809    6506 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023814    6506 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023819    6506 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023823    6506 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.023828    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.029081    6506 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.029101    6506 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.029130    6506 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.029135    6506 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029196    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:03:06.040605 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029201    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029204    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029207    6506 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029210    6506 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029212    6506 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029215    6506 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029219    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029224    6506 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029229    6506 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029233    6506 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029238    6506 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029242    6506 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029246    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029251    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029259    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029263    6506 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029265    6506 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029268    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029271    6506 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:03:06.041302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029274    6506 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029276    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029278    6506 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029281    6506 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029284    6506 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029286    6506 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029289    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029292    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029294    6506 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029296    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029299    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029302    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029304    6506 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029307    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029310    6506 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029312    6506 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029315    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029317    6506 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029319    6506 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029322    6506 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:03:06.042003 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029324    6506 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029326    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029329    6506 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029331    6506 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029333    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029336    6506 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029338    6506 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029341    6506 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029343    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029346    6506 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029349    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029351    6506 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029354    6506 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029356    6506 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029359    6506 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029361    6506 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029364    6506 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029366    6506 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029369    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029371    6506 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:03:06.042835 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029373    6506 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029376    6506 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029378    6506 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029381    6506 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029384    6506 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029386    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029389    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029391    6506 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029393    6506 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029396    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029398    6506 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029401    6506 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029404    6506 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029408    6506 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029411    6506 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029415    6506 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029419    6506 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029422    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029425    6506 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:03:06.043401 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029428    6506 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029431    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029434    6506 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029437    6506 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029440    6506 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029443    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029445    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029448    6506 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029450    6506 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029453    6506 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029455    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029458    6506 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029460    6506 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029463    6506 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029465    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029467    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029470    6506 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029472    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029475    6506 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:03:06.044000 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029477    6506 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029480    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029482    6506 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029484    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029487    6506 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029489    6506 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029492    6506 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029494    6506 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.029521    6506 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029664    6506 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029670    6506 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029673    6506 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029676    6506 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029679    6506 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029682    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029685    6506 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:03:06.044664 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029688    6506 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029690    6506 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029693    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029695    6506 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029698    6506 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029701    6506 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029705    6506 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029709    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029712    6506 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029714    6506 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029717    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029719    6506 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029722    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029724    6506 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029727    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029729    6506 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029731    6506 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029734    6506 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029736    6506 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029739    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:03:06.045133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029741    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029744    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029746    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029748    6506 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029751    6506 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029753    6506 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029760    6506 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029762    6506 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029765    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029768    6506 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029770    6506 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029774    6506 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029777    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029780    6506 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029782    6506 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029785    6506 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029787    6506 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029790    6506 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029811    6506 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:03:06.045776 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029814    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029817    6506 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029819    6506 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029822    6506 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029824    6506 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029827    6506 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029830    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029833    6506 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029835    6506 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029838    6506 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029840    6506 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029843    6506 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029845    6506 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029848    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029850    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029853    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029855    6506 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029858    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029860    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:03:06.046417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029865    6506 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029869    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029872    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029874    6506 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029877    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029879    6506 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029883    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029885    6506 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029888    6506 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029890    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029893    6506 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029895    6506 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029898    6506 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029900    6506 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029903    6506 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029905    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029908    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029910    6506 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029913    6506 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029915    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:03:06.047059 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029917    6506 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029920    6506 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029923    6506 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029925    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029928    6506 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029930    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029933    6506 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029935    6506 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029938    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029940    6506 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029942    6506 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029945    6506 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029948    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029951    6506 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029954    6506 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029957    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029959    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029962    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029964    6506 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029968    6506 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:03:06.047647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.029970    6506 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.029994    6506 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030026    6506 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030031    6506 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030034    6506 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030037    6506 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030040    6506 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030061    6506 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030064    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030067    6506 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030070    6506 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030073    6506 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030075    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030078    6506 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030080    6506 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030084    6506 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:03:06.048234 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030087    6506 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030090    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030093    6506 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030096    6506 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030098    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030101    6506 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030103    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030106    6506 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030108    6506 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030112    6506 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030115    6506 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030117    6506 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030120    6506 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030122    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030125    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030128    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030130    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030133    6506 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030136    6506 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030138    6506 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:03:06.048665 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030141    6506 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030143    6506 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030146    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030149    6506 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030151    6506 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030154    6506 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030156    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030159    6506 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030161    6506 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030164    6506 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030166    6506 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030169    6506 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030171    6506 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030174    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030177    6506 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030179    6506 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030181    6506 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030184    6506 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030186    6506 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030189    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:03:06.049349 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030191    6506 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030195    6506 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030197    6506 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030200    6506 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030202    6506 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030205    6506 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030208    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030210    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030213    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030216    6506 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030219    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030221    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030225    6506 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030228    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030230    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030233    6506 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030235    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030238    6506 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030240    6506 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:03:06.050155 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030243    6506 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030245    6506 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030248    6506 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030250    6506 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030253    6506 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030255    6506 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030258    6506 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030260    6506 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030263    6506 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030266    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030268    6506 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030271    6506 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030273    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030276    6506 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030280    6506 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030282    6506 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030284    6506 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030287    6506 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030289    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030292    6506 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:03:06.050666 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030295    6506 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030297    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030299    6506 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030303    6506 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030305    6506 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030310    6506 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030312    6506 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030315    6506 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030318    6506 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030320    6506 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030324    6506 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030326    6506 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:06.030329    6506 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.030378    6506 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.031307    6506 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.031364    6506 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.031553    6506 server.go:1014] "Starting client certificate rotation"
Nov 03 16:03:06.051209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.031689    6506 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 16:03:06.051698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.031739    6506 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 15:56:54 +0000 UTC" deadline="2028-04-07 16:28:47.918303278 +0000 UTC"
Nov 03 16:03:06.051698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.031766    6506 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12504h25m41.886539093s"
Nov 03 16:03:06.051698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.032089    6506 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 16:03:06.051698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.033139    6506 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 16:03:06.051698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.034547    6506 log.go:25] "Validated CRI v1 runtime API"
Nov 03 16:03:06.051698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.036511    6506 log.go:25] "Validated CRI v1 image API"
Nov 03 16:03:06.051698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.037314    6506 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 16:03:06.051698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.039612    6506 fs.go:135] Filesystem UUIDs: map[4d2cae4f-9a11-440d-95af-4c8739cc81ee:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c0570534-d3c5-47dd-9c86-ebbf3feea646:/dev/nvme0n1p3]
Nov 03 16:03:06.051950 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.039626    6506 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0154a7620c880551c3e2fd70a07c4c1a3737e2d53f1bf32fd621f3996dd989e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0154a7620c880551c3e2fd70a07c4c1a3737e2d53f1bf32fd621f3996dd989e7/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b60c8b03a2166e60967970a0a532a3e32e619e8c11a006d43b2aed5cb1f2e67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b60c8b03a2166e60967970a0a532a3e32e619e8c11a006d43b2aed5cb1f2e67/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c1e5ce477e74f85116c5361dffe10a499ce6966005b6620de86c37ed3734ae9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c1e5ce477e74f85116c5361dffe10a499ce6966005b6620de86c37ed3734ae9/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46959e3250cc1eaada8d5b775cc0246c475b65a4363584dc4836bb5ac1dd8534/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46959e3250cc1eaada8d5b775cc0246c475b65a4363584dc4836bb5ac1dd8534/userdata/shm major:0 minor:297 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b6a95384b4a90b0adf8bdd0aae714ad41949c944146414374807e9dc4c93a4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b6a95384b4a90b0adf8bdd0aae714ad41949c944146414374807e9dc4c93a4e/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/605cfa2f89d3ab39191e9507716386203ecebddfbe129f696ff146970c375629/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/605cfa2f89d3ab39191e9507716386203ecebddfbe129f696ff146970c375629/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d99e95fb1a77cc650cf6ea4ec19dc3847fe12e58e81982442fe06ff7415c687/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d99e95fb1a77cc650cf6ea4ec19dc3847fe12e58e81982442fe06ff7415c687/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fee9c50afd4eae2291676cb5f4db803da77aacba92ec62493abd5e3be400be8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fee9c50afd4eae2291676cb5f4db803da77aacba92ec62493abd5e3be400be8/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78598f65fa6cf2cae3f644a79a9eeb9e7b0eb3c5b355da0b32c5cb9a11c619b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78598f65fa6cf2cae3f644a79a9eeb9e7b0eb3c5b355da0b32c5cb9a11c619b3/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78a6603de7ff47bc15888ece6c1728a5f9cae64f61d8a7b06abb4c1785c1b56e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78a6603de7ff47bc15888ece6c1728a5f9cae64f61d8a7b06abb4c1785c1b56e/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7da326ffd374624f716d47a5caad64927991919e4cfc64e5a6377275609f4107/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7da326ffd374624f716d47a5caad64927991919e4cfc64e5a6377275609f4107/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1fe67ccc0ffbc19e2e96cbc485098416398466cf149c1884a554c830799fa00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1fe67ccc0ffbc19e2e96cbc485098416398466cf149c1884a554c830799fa00/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afdc0a4848ba4fbb0befd56b1d5712ec0b6d92fe8703fbd426a5962594b34a04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afdc0a4848ba4fbb0befd56b1d5712ec0b6d92fe8703fbd426a5962594b34a04/userdata/shm major:0 minor:398 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9c0ea4d51039c6b39d99d735c2078885371f2ab3771a3b92a44db93c6c9cff4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9c0ea4d51039c6b39d99d735c2078885371f2ab3771a3b92a44db93c6c9cff4/userdata/shm major:0 minor:256 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf3d135781ee7be6c7d6f89592858a5911d2ea65f7ab1379bab219ec9134cb74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf3d135781ee7be6c7d6f89592858a5911d2ea65f7ab1379bab219ec9134cb74/userdata/shm major:0 minor:396 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9460f4260a6754bf5741b5addc46c13a81142ddf334c3010eae472814c46844/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9460f4260a6754bf5741b5addc46c13a81142ddf334c3010eae472814c46844/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd13036ac6e4a5fa595a87799d6e14ccb8b2b8e2c23890c9d58ed3889eb7ed26/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd13036ac6e4a5fa595a87799d6e14ccb8b2b8e2c23890c9d58ed3889eb7ed26/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0fb0377af2f674020219c26fa4a1f42c72cea83140ef2909ceca1f643b17f94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0fb0377af2f674020219c26fa4a1f42c72cea83140ef2909ceca1f643b17f94/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfb883769f66b407fc016c03dffa3a4dc0231923b0820cb721c0aae7d6e71ff6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfb883769f66b407fc016c03dffa3a4dc0231923b0820cb721c0aae7d6e71ff6/userdata/shm major:0 minor:307 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f42d395a33f328559930baf996d8efe1eb704477cd5e0fc327ac3581fd0bf5cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f42d395a33f328559930baf996d8efe1eb704477cd5e0fc327ac3581fd0bf5cd/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f75f297c8ad68d2c4a0c864db203eb147ee523fd6577e564d6462907f853962a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f75f297c8ad68d2c4a0c864db203eb147ee523fd6577e564d6462907f853962a/userdata/shm major:0 minor:400 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13d056b1-85f8-48eb-b9bf-aeb4b98a8170/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/13d056b1-85f8-48eb-b9bf-aeb4b98a8170/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ece33e-8a7b-418f-aa9e-eaa2fed48273/volumes/kubernetes.io~projected/kube-api-access-rs5sr:{mountpoint:/var/lib/kubelet/pods/13ece33e-8a7b-418f-aa9e-eaa2fed48273/volumes/kubernetes.io~projected/kube-api-access-rs5sr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c44c5cc-7fb0-40c7-970b-add5c87c0770/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/1c44c5cc-7fb0-40c7-970b-add5c87c0770/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c44c5cc-7fb0-40c7-970b-add5c87c0770/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/1c44c5cc-7fb0-40c7-970b-add5c87c0770/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c44c5cc-7fb0-40c7-970b-add5c87c0770/volumes/kubernetes.io~projected/kube-api-access-wwxn6:{mountpoint:/var/lib/kubelet/pods/1c44c5cc-7fb0-40c7-970b-add5c87c0770/volumes/kubernetes.io~projected/kube-api-access-wwxn6 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1dbf1e42-f480-47a4-8382-6207e2fe7774/volumes/kubernetes.io~projected/kube-api-access-xn24w:{mountpoint:/var/lib/kubelet/pods/1dbf1e42-f480-47a4-8382-6207e2fe7774/volumes/kubernetes.io~projected/kube-api-access-xn24w major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e20bd59-e885-4662-bb6b-7cd403b9d752/volumes/kubernetes.io~projected/kube-api-access-fw9h4:{mountpoint:/var/lib/kubelet/pods/1e20bd59-e885-4662-bb6b-7cd403b9d752/volumes/kubernetes.io~projected/kube-api-access-fw9h4 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e20bd59-e885-4662-bb6b-7cd403b9d752/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/1e20bd59-e885-4662-bb6b-7cd403b9d752/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e20bd59-e885-4662-bb6b-7cd403b9d752/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/1e20bd59-e885-4662-bb6b-7cd403b9d752/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e20bd59-e885-4662-bb6b-7cd403b9d752/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/1e20bd59-e885-4662-bb6b-7cd403b9d752/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7/volumes/kubernetes.io~projected/kube-api-access-v7s62:{mountpoint:/var/lib/kubelet/pods/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7/volumes/kubernetes.io~projected/kube-api-access-v7s62 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5806796f-e98d-4eb1-b24c-d4c0cfc057f4/volumes/kubernetes.io~projected/kube-api-access-krj9t:{mountpoint:/var/lib/kubelet/pods/5806796f-e98d-4eb1-b24c-d4c0cfc057f4/volumes/kubernetes.io~projected/kube-api-access-krj9t major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5806796f-e98d-4eb1-b24c-d4c0cfc057f4/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/5806796f-e98d-4eb1-b24c-d4c0cfc057f4/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volume-subpaths/web-config/config-reloader/2 major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volume-subpaths/web-config/prometheus/15 major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~empty-dir/config-out major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~projected/kube-api-access-kfqjb:{mountpoint:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~projected/kube-api-access-kfqjb major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~projected/tls-assets major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/config major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/web-config major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/596632c1-fb08-4008-ad12-7d399fa3994b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/596632c1-fb08-4008-ad12-7d399fa3994b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/596632c1-fb08-4008-ad12-7d399fa3994b/volumes/kubernetes.io~projected/kube-api-access-svwtn:{mountpoint:/var/lib/kubelet/pods/596632c1-fb08-4008-ad12-7d399fa3994b/volumes/kubernetes.io~projected/kube-api-access-svwtn major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/596632c1-fb08-4008-ad12-7d399fa3994b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/596632c1-fb08-4008-ad12-7d399fa3994b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61353437-1060-48e2-89f0-b51beb1fdb60/volumes/kubernetes.io~projected/kube-api-access-66f87:{mountpoint:/var/lib/kubelet/pods/61353437-1060-48e2-89f0-b51beb1fdb60/volumes/kubernetes.io~projected/kube-api-access-66f87 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61353437-1060-48e2-89f0-b51beb1fdb60/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/61353437-1060-48e2-89f0-b51beb1fdb60/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ea5065d-e1c6-4152-842c-833cd8155772/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7ea5065d-e1c6-4152-842c-833cd8155772/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c83af76-9520-4c3d-8aec-3590c97abaf7/volumes/kubernetes.io~projected/kube-api-access-q5r8x:{mountpoint:/var/lib/kubelet/pods/9c83af76-9520-4c3d-8aec-3590c97abaf7/volumes/kubernetes.io~projected/kube-api-access-q5r8x major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c83af76-9520-4c3d-8aec-3590c97abaf7/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9c83af76-9520-4c3d-8aec-3590c97abaf7/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c83af76-9520-4c3d-8aec-3590c97abaf7/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/9c83af76-9520-4c3d-8aec-3590c97abaf7/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f0811b4-9b91-454b-b715-6aaf519b5e92/volumes/kubernetes.io~projected/kube-api-access-748gh:{mountpoint:/var/lib/kubelet/pods/9f0811b4-9b91-454b-b715-6aaf519b5e92/volumes/kubernetes.io~projected/kube-api-access-748gh major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f0811b4-9b91-454b-b715-6aaf519b5e92/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9f0811b4-9b91-454b-b715-6aaf519b5e92/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f0811b4-9b91-454b-b715-6aaf519b5e92/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/9f0811b4-9b91-454b-b715-6aaf519b5e92/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a00bade6-34ba-4d87-a740-9727b8372650/volumes/kubernetes.io~projected/kube-api-access-9hsrw:{mountpoint:/var/lib/kubelet/pods/a00bade6-34ba-4d87-a740-9727b8372650/volumes/kubernetes.io~projected/kube-api-access-9hsrw major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a00bade6-34ba-4d87-a740-9727b8372650/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/a00bade6-34ba-4d87-a740-9727b8372650/volumes/kubernetes.io~secret/metrics-certs major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9f7db5a-c0f7-480e-a19f-8b77e7285af2/volumes/kubernetes.io~projected/kube-api-access-24wql:{mountpoint:/var/lib/kubelet/pods/a9f7db5a-c0f7-480e-a19f-8b77e7285af2/volumes/kubernetes.io~projected/kube-api-access-24wql major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9f7db5a-c0f7-480e-a19f-8b77e7285af2/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a9f7db5a-c0f7-480e-a19f-8b77e7285af2/volumes/kubernetes.io~secret/cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af63014e-5ed6-4213-8838-529749c931cb/volumes/kubernetes.io~projected/kube-api-access-r4774:{mountpoint:/var/lib/kubelet/pods/af63014e-5ed6-4213-8838-529749c931cb/volumes/kubernetes.io~projected/kube-api-access-r4774 major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0/volumes/kubernetes.io~projected/kube-api-access-x9jz2:{mountpoint:/var/lib/kubelet/pods/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0/volumes/kubernetes.io~projected/kube-api-access-x9jz2 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0652c5d-5440-4b54-9220-fbf77c06ec2f/volumes/kubernetes.io~projected/kube-api-access-8qrc5:{mountpoint:/var/lib/kubelet/pods/c0652c5d-5440-4b54-9220-fbf77c06ec2f/volumes/kubernetes.io~projected/kube-api-access-8qrc5 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~projected/kube-api-access-pfqzs:{mountpoint:/var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~projected/kube-api-access-pfqzs major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998/volumes/kubernetes.io~projected/kube-api-access-qkcbp:{mountpoint:/var/lib/kubelet/pods/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998/volumes/kubernetes.io~projected/kube-api-access-qkcbp major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/fdf8f5e1ccd2eaa583f2379789053581564adb59528b32a65bc169a1ce1dc170/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2e9c342973d7d54d5eb54c998fe759eea4da382ddace71b0c1f63b7cb7b4cb7f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b8337c2dd333cc1961c31b38f7446e994fe26d2a89f48433ed17c1ba60b86441/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/4df20064a39c1ee62fd4a70b4755ef7bd7efe173ed427d1d5574d3822bc0a5c2/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/20c7b11e7c1df442864f1333843e920fb4fa4df7c9263cffaae1a301508dc44a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5bfcb405330e6d1d4eee7ed245cf14b4099fa2e6eb52c55ddfc72cc6d975e20f/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1525915883c61fd66be8927290613a9087e37a1ac38aefeb24947367638f78c8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/06a324e62e38b09c3f6843dba4f3f5d3465a839d6e5614ca401d30d12dd04144/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3d7a670343556abaa6c4bc9c02ede9df1dff4f78f76ccdcc516aa6333fdfdcf9/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/e6cb5dd38785ec061bf74389ab433d2b10aaa2af6f96a4b77d019727731b431d/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/6f2688870f0c49e76c20c2ebb8a63020e1289766552b7ccf1dad41adab86dc60/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/effb1d50ea59b281aa48408db231069b5eb8d3df618a5f88fc2ad718c471a4ee/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/27c2a27e4795dcc6e47aff721625495286ae0ee64804f7433415e66bce52ad43/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/405f288d7b025f0c3915866371ac52b454c3ad42285f37f21aa85934ab331e98/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ed7f29d53f43db7366729ca8a5616ad9a269b47d7b8ffc8fd135e85e8f23887c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e675c220885694e9cf061d9422b96c0e79906d70130bacb43145d5e1c461fe1b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/7bbcbb8efa28e899aae64f7690a17a90251a18235c074a8ba41a2a475e8e3cb4/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/b89a5146f997822a7b636cd9e69c1d0aa6bb62c6d25f75e9ab43fd0b2e03b69d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/6424cdeed941cb43f88199b94dd0f4895a8bee809966d545d36af9d25025e441/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/e23c693b009fb44c063439fe11df54bcf3806c29cea98f280807827863b3df25/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/16c9feba3eb328fd543d34ff9717e911bd408b165b95df4e8786e7033352103d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/405cc59ded9b80d68fcb960b06863c2c0af43b159c324958b26afa7e54c9a29f/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/04f1f1ad7b6c96912420725cbfb77815767c614dd3477d1a0a52fe635d8bdb26/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/5a5c2d2d2618b3e8a00d78914d768fb12310230a1902edf26bd7564f0aee4f44/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/6f2da3d04dcacbf95b657443d14cea4d6ee55ba9adf9c4281f98c763aca80e63/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/40d7b4f4df99c08d044ee641662d09269db638e329350999cdb0f8d5f904a1d7/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/da4f8848938bea904b0ba6ffe04283a1f00d562e2acb755925faee54d87d4d63/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/868afe7ab8b0003a52028da514b6ef00a16df1ca3c0b71055d98963383e28671/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/e4e203580916df27693147fdbb0f5a2347063460ce9701a8396053011b6c35c3/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/4e5d07744ab60279e66899e55f46bcac824d05760de2448e7026af1c179b9897/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/46d019cdb4b8506557239ab91a02456e8869b8934607cb6ac671e3b76df18450/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/24cf93d0886469b9aac8b5388fbd4a289d43ead6cce38185af100d3024490657/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/c373ef850b4632576979406ac0440c538cf3da4bcdcf8fc50b07a20bca0341f4/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/f7462c7afd91d4ebfde1541f56d4fb8c3721efcc828e1a28fe827ade26c8f686/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/978ebb327b816f3331a21ce1f05fa37ceca1c00b6c1b33c74d3f2480f5e2bc0c/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/a029479e9ba2a8b76e1cec54ddbaaa062047d9421dfd52a6df492e4d69bf1a4e/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/7a08e5bdb719e46e72f57165efc5427e1fe5ea12077746672e43b018c492c10d/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/3b8e64ef064db45e880f12234cb9cb656f3d157e5b828d1145564a1dea01c892/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/f94d6a4c1714ea789b338c4d7952fd26aac7b5fb17d21e2450496fe175b05104/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/1fd5540a3087e13509ffbb5e7b6f6d66895149d744ac5304b5eed2ba3432c1aa/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/886eb8eed9a3c5926c8e36584d3fd8ffa7515f95f1e1e4c61c74192279ef2a1b/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/b82c3ac21cd8b65bb2d56a4903a1c7afbf3d846cbb9c651ae62b4e72db59ce8d/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/c741fdfa507aa4aeecdee4a8a349b4c18c3c5a8a6499252d298e279eaa7c18b0/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/325ab185ba3f537beb1fb6c63535b87450b4c53a6533b5a4a2455a4652ba982e/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/f9b9bca7c388dcb407af52b3ad2e2b06d889016271fbc31d318c080c075ca7e3/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/1cbc64a7b7aa262ab5f8b26500c6164655fb0d83fe0a479190e17b09c6dc6c68/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/d7c3b09e8765a823fd82c01564363de435fa27d4fb430f9d62d25e4f3501ad31/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/0aa686b82a4ce5af6c7ebdc606ec283bd607c847c5bb219e7f12872268ca37c4/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/fb34119b4a9bd1fe6639f5e86af713427053add87430b64c813e2d023b2a22e3/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/0e6a36e158cd703ed0fafcaef7e32f61cb11e4347fabb510f9d44c67f2c95194/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/800eac045a568f1ff74a8efb5aab5a0c6ab81841218d67c7d0e2adc122530a46/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/c02e675403413052f29d513dc64f3be69dfb2a7b159584f98c09ad4c8d2f870a/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/af560766c963f4b0ce7ad8e2559811db90b1f280b3b3be191231fff111477957/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/084c3db74d857650a655ff02a1868b88ca7551e3bd4803600f78a91de2b28f7a/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/c2e944f10e21a50c5c61f2248a2dbe61cda38018c73803e48d6c1935ba530127/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/b618d55609939c4260b2bf250c0f182eeada3188e3e993a6640c301e8dee1f7b/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/3c8e162e9770e917438917e23c01d379c00b3f3531754713d3fa8bbd6b60fae3/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/c9ae9778a126e5acdff0928b3d87e60bdc213047a37728b5bb8dd758d49dafc6/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/2a59a3827ca032c69b8ba4a398e3b33dc255ed9841901696af78d9c065fae97e/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/e3d030448f4ab98ba91d93ded0827e067b8773b69b0f81f9535db1a5f85c8708/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/cf91c2762762132aba9d48842cd9e8aa87908baa0032e3761c312b286748f96f/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/91827898377e2f69ae5fa6440e7bb685fae0921415bb21aee18ff563a4ebf5cf/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/17a56df824bdb8e3a0f6c1fa5c08c7083b43133a68312fb8dc3bc9466d653345/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/8ac999e509b9c5fc0e6eb19671f787f0fc671fb5f14b95926fd90c8b6b841a34/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/75b4776b10074de4cf79e64d597d36a5be88a14fe26d7d5dc47c10131b3d0be0/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/e48615315eebe9d9455c53ddb63bfbe69f4ddc20575630c192dce663c466bd3c/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/37d845b7398f24e70ffb0ba6eb8d6f01522c96b46669b57e755d9dcd5d575f87/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/a988fe42f0720a633788f4b5e1ab96fc5ed04db33c7d3640d3df6ac669b00a13/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/4ba6d517f91e824434ecc3523abf7056fe6c03796292390733910ca3e89c0818/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ba333e9c5f610a71945af1ddc1e7021078cb3693670a4cef7cba045d8a9442f2/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.047645    6506 manager.go:217] Machine: {Timestamp:2026-11-03 16:03:06.047175667 +0000 UTC m=+0.060573805 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100483 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2358ba4325e67b13944d012380d40a SystemUUID:ec2358ba-4325-e67b-1394-4d012380d40a BootID:8b91d1e5-6aa7-4810-87c6-fdcf00a77ede Filesystems:[{Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:342 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9f7db5a-c0f7-480e-a19f-8b77e7285af2/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:345 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1dbf1e42-f480-47a4-8382-6207e2fe7774/volumes/kubernetes.io~projected/kube-api-access-xn24w DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/596632c1-fb08-4008-ad12-7d399fa3994b/volumes/kubernetes.io~projected/kube-api-access-svwtn DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:343 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c1e5ce477e74f85116c5361dffe10a499ce6966005b6620de86c37ed3734ae9/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13ece33e-8a7b-418f-aa9e-eaa2fed48273/volumes/kubernetes.io~projected/kube-api-access-rs5sr DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f42d395a33f328559930baf996d8efe1eb704477cd5e0fc327ac3581fd0bf5cd/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78a6603de7ff47bc15888ece6c1728a5f9cae64f61d8a7b06abb4c1785c1b56e/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afdc0a4848ba4fbb0befd56b1d5712ec0b6d92fe8703fbd426a5962594b34a04/userdata/shm DeviceMajor:0 DeviceMinor:398 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5806796f-e98d-4eb1-b24c-d4c0cfc057f4/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:337 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:346 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:347 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:294 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b6a95384b4a90b0adf8bdd0aae714ad41949c944146414374807e9dc4c93a4e/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c83af76-9520-4c3d-8aec-3590c97abaf7/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:347 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~projected/kube-api-access-kfqjb DeviceMajor:0 DeviceMinor:351 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:291 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c44c5cc-7fb0-40c7-970b-add5c87c0770/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:347 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/af63014e-5ed6-4213-8838-529749c931cb/volumes/kubernetes.io~projected/kube-api-access-r4774 DeviceMajor:0 DeviceMinor:395 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fee9c50afd4eae2291676cb5f4db803da77aacba92ec62493abd5e3be400be8/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9c0ea4d51039c6b39d99d735c2078885371f2ab3771a3b92a44db93c6c9cff4/userdata/shm DeviceMajor:0 DeviceMinor:256 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7/volumes/kubernetes.io~projected/kube-api-access-v7s62 DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf3d135781ee7be6c7d6f89592858a5911d2ea65f7ab1379bab219ec9134cb74/userdata/shm DeviceMajor:0 DeviceMinor:396 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5806796f-e98d-4eb1-b24c-d4c0cfc057f4/volumes/kubernetes.io~projected/kube-api-access-krj9t DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e20bd59-e885-4662-bb6b-7cd403b9d752/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:302 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:349 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13d056b1-85f8-48eb-b9bf-aeb4b98a8170/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998/volumes/kubernetes.io~projected/kube-api-access-qkcbp DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a9f7db5a-c0f7-480e-a19f-8b77e7285af2/volumes/kubernetes.io~projected/kube-api-access-24wql DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46959e3250cc1eaada8d5b775cc0246c475b65a4363584dc4836bb5ac1dd8534/userdata/shm DeviceMajor:0 DeviceMinor:297 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:341 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7ea5065d-e1c6-4152-842c-833cd8155772/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:394 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/61353437-1060-48e2-89f0-b51beb1fdb60/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1e20bd59-e885-4662-bb6b-7cd403b9d752/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:303 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f75f297c8ad68d2c4a0c864db203eb147ee523fd6577e564d6462907f853962a/userdata/shm DeviceMajor:0 DeviceMinor:400 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f0811b4-9b91-454b-b715-6aaf519b5e92/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:246 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9c83af76-9520-4c3d-8aec-3590c97abaf7/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:253 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:344 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0fb0377af2f674020219c26fa4a1f42c72cea83140ef2909ceca1f643b17f94/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9460f4260a6754bf5741b5addc46c13a81142ddf334c3010eae472814c46844/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd13036ac6e4a5fa595a87799d6e14ccb8b2b8e2c23890c9d58ed3889eb7ed26/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c0652c5d-5440-4b54-9220-fbf77c06ec2f/volumes/kubernetes.io~projected/kube-api-access-8qrc5 DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/61353437-1060-48e2-89f0-b51beb1fdb60/volumes/kubernetes.io~projected/kube-api-access-66f87 DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:293 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:295 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:348 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/596632c1-fb08-4008-ad12-7d399fa3994b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f0811b4-9b91-454b-b715-6aaf519b5e92/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:252 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfb883769f66b407fc016c03dffa3a4dc0231923b0820cb721c0aae7d6e71ff6/userdata/shm DeviceMajor:0 DeviceMinor:307 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c44c5cc-7fb0-40c7-970b-add5c87c0770/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a00bade6-34ba-4d87-a740-9727b8372650/volumes/kubernetes.io~projected/kube-api-access-9hsrw DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d99e95fb1a77cc650cf6ea4ec19dc3847fe12e58e81982442fe06ff7415c687/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9c83af76-9520-4c3d-8aec-3590c97abaf7/volumes/kubernetes.io~projected/kube-api-access-q5r8x DeviceMajor:0 DeviceMinor:255 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:292 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/1c44c5cc-7fb0-40c7-970b-add5c87c0770/volumes/kubernetes.io~projected/kube-api-access-wwxn6 DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0154a7620c880551c3e2fd70a07c4c1a3737e2d53f1bf32fd621f3996dd989e7/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7da326ffd374624f716d47a5caad64927991919e4cfc64e5a6377275609f4107/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cc926104-1c6c-46f3-9d8a-7f1d8394d713/volumes/kubernetes.io~projected/kube-api-access-pfqzs DeviceMajor:0 DeviceMinor:296 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1fe67ccc0ffbc19e2e96cbc485098416398466cf149c1884a554c830799fa00/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b60c8b03a2166e60967970a0a532a3e32e619e8c11a006d43b2aed5cb1f2e67/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78598f65fa6cf2cae3f644a79a9eeb9e7b0eb3c5b355da0b32c5cb9a11c619b3/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e20bd59-e885-4662-bb6b-7cd403b9d752/volumes/kubernetes.io~projected/kube-api-access-fw9h4 DeviceMajor:0 DeviceMinor:304 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:347 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/596632c1-fb08-4008-ad12-7d399fa3994b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0/volumes/kubernetes.io~projected/kube-api-access-x9jz2 DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f0811b4-9b91-454b-b715-6aaf519b5e92/volumes/kubernetes.io~projected/kube-api-access-748gh DeviceMajor:0 DeviceMinor:254 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/605cfa2f89d3ab39191e9507716386203ecebddfbe129f696ff146970c375629/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e20bd59-e885-4662-bb6b-7cd403b9d752/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:301 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a00bade6-34ba-4d87-a740-9727b8372650/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:390 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:350 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3c1e5ce477e74f8 MacAddress:f6:9d:64:c4:3e:2b Speed:10000 Mtu:8901} {Name:46959e3250cc1ea MacAddress:b6:d6:10:c7:e9:68 Speed:10000 Mtu:8901} {Name:605cfa2f89d3ab3 MacAddress:92:e2:9a:f0:02:c6 Speed:10000 Mtu:8901} {Name:78a6603de7ff47b MacAddress:52:64:87:dd:8d:44 Speed:10000 Mtu:8901} {Name:afdc0a4848ba4fb MacAddress:2a:b9:79:42:e8:a4 Speed:10000 Mtu:8901} {Name:bf3d135781ee7be MacAddress:aa:7a:ca:b5:10:f0 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:da:17:71:49 Speed:0 Mtu:9001} {Name:br-int MacAddress:e6:27:80:3e:f9:81 Speed:0 Mtu:8901} {Name:c9460f4260a6754 MacAddress:9e:92:13:2f:92:9f Speed:10000 Mtu:8901} {Name:cd13036ac6e4a5f MacAddress:a2:66:b2:5e:db:a1 Speed:10000 Mtu:8901} {Name:dfb883769f66b40 MacAddress:16:18:d1:56:94:62 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:da:17:71:49 Speed:0 Mtu:9001} {Name:f75f297c8ad68d2 MacAddress:06:04:18:3f:02:b0 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:0a:73:d5:53:b4:35 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a2:fb:16:1d:65:5f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.048047    6506 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.048093    6506 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.048410    6506 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.048432    6506 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-159.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.048558    6506 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.048565    6506 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.048578    6506 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.048598    6506 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.048674    6506 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.048841    6506 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.048952    6506 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.048986    6506 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.048997    6506 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.049007    6506 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.049015    6506 kubelet.go:405] "Adding apiserver pod source"
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.049023    6506 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.049776    6506 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.049789    6506 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 16:03:06.052191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.050313    6506 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 16:03:06.052928 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.052862    6506 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 16:03:06.052928 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.052871    6506 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 16:03:06.052979 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.052965    6506 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 16:03:06.053160 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053148    6506 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 16:03:06.053193 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053177    6506 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 16:03:06.053275 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053266    6506 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 16:03:06.053306 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053280    6506 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 16:03:06.053306 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053286    6506 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 16:03:06.053306 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053292    6506 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 16:03:06.053306 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053307    6506 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 16:03:06.053407 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053313    6506 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 16:03:06.053407 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053319    6506 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 16:03:06.053407 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053327    6506 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 16:03:06.053407 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053337    6506 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 16:03:06.053407 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053344    6506 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 16:03:06.053407 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053359    6506 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 16:03:06.053407 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053368    6506 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 16:03:06.053407 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053394    6506 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 16:03:06.053407 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.053403    6506 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 16:03:06.057068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.057052    6506 server.go:1263] "Started kubelet"
Nov 03 16:03:06.057282 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.057210    6506 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 16:03:06.057543 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.057294    6506 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 16:03:06.057570 ip-10-0-137-159 systemd[1]: Started Kubernetes Kubelet.
Nov 03 16:03:06.058343 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.057199    6506 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 16:03:06.059452 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.059418    6506 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 16:03:06.061233 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.061216    6506 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 16:03:06.062693 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.062675    6506 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 16:03:06.063207 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.063080    6506 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 16:03:06.063207 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.063094    6506 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 16:03:06.063207 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.063111    6506 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 16:03:06.063909 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:06.063870    6506 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-159.us-east-2.compute.internal\" not found"
Nov 03 16:03:06.064139 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.063097    6506 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 16:03:06.064139 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.064069    6506 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:56:56 +0000 UTC" deadline="2028-07-25 09:50:11.234453108 +0000 UTC"
Nov 03 16:03:06.064139 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.064097    6506 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15113h47m5.170358702s"
Nov 03 16:03:06.067144 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.067054    6506 factory.go:55] Registering systemd factory
Nov 03 16:03:06.067144 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.067077    6506 factory.go:223] Registration of the systemd container factory successfully
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068501    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f0811b4-9b91-454b-b715-6aaf519b5e92" volumeName="kubernetes.io/configmap/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068539    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc926104-1c6c-46f3-9d8a-7f1d8394d713" volumeName="kubernetes.io/configmap/cc926104-1c6c-46f3-9d8a-7f1d8394d713-metrics-client-ca" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068543    6506 factory.go:153] Registering CRI-O factory
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068554    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c44c5cc-7fb0-40c7-970b-add5c87c0770" volumeName="kubernetes.io/empty-dir/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-tuned" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068562    6506 factory.go:223] Registration of the crio container factory successfully
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068574    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e20bd59-e885-4662-bb6b-7cd403b9d752" volumeName="kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068587    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068600    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c83af76-9520-4c3d-8aec-3590c97abaf7" volumeName="kubernetes.io/secret/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-tls" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068611    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3b0f6f7-b58c-49a1-aa66-57cee4ae2998" volumeName="kubernetes.io/projected/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-kube-api-access-qkcbp" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068625    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e20bd59-e885-4662-bb6b-7cd403b9d752" volumeName="kubernetes.io/projected/1e20bd59-e885-4662-bb6b-7cd403b9d752-kube-api-access-fw9h4" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068642    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7" volumeName="kubernetes.io/projected/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-kube-api-access-v7s62" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068656    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068667    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="596632c1-fb08-4008-ad12-7d399fa3994b" volumeName="kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-ovnkube-config" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068616    6506 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068705    6506 factory.go:103] Registering Raw factory
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068719    6506 manager.go:1199] Started watching for new ooms in manager
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068679    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61353437-1060-48e2-89f0-b51beb1fdb60" volumeName="kubernetes.io/secret/61353437-1060-48e2-89f0-b51beb1fdb60-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068859    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f0811b4-9b91-454b-b715-6aaf519b5e92" volumeName="kubernetes.io/configmap/9f0811b4-9b91-454b-b715-6aaf519b5e92-metrics-client-ca" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068883    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f0811b4-9b91-454b-b715-6aaf519b5e92" volumeName="kubernetes.io/secret/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068901    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc926104-1c6c-46f3-9d8a-7f1d8394d713" volumeName="kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068915    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ece33e-8a7b-418f-aa9e-eaa2fed48273" volumeName="kubernetes.io/projected/13ece33e-8a7b-418f-aa9e-eaa2fed48273-kube-api-access-rs5sr" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068928    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e20bd59-e885-4662-bb6b-7cd403b9d752" volumeName="kubernetes.io/configmap/1e20bd59-e885-4662-bb6b-7cd403b9d752-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068942    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5806796f-e98d-4eb1-b24c-d4c0cfc057f4" volumeName="kubernetes.io/projected/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-kube-api-access-krj9t" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068956    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068968    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f0811b4-9b91-454b-b715-6aaf519b5e92" volumeName="kubernetes.io/projected/9f0811b4-9b91-454b-b715-6aaf519b5e92-kube-api-access-748gh" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068983    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baadfd63-5ceb-48e0-a5c4-c3d2138e18c0" volumeName="kubernetes.io/projected/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-kube-api-access-x9jz2" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.068996    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0652c5d-5440-4b54-9220-fbf77c06ec2f" volumeName="kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069008    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc926104-1c6c-46f3-9d8a-7f1d8394d713" volumeName="kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069021    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c44c5cc-7fb0-40c7-970b-add5c87c0770" volumeName="kubernetes.io/empty-dir/1c44c5cc-7fb0-40c7-970b-add5c87c0770-tmp" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069037    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dbf1e42-f480-47a4-8382-6207e2fe7774" volumeName="kubernetes.io/configmap/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-daemon-config" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069051    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069067    6506 manager.go:319] Starting recovery of all containers
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069063    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-db" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069097    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069116    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a00bade6-34ba-4d87-a740-9727b8372650" volumeName="kubernetes.io/projected/a00bade6-34ba-4d87-a740-9727b8372650-kube-api-access-9hsrw" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069129    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9f7db5a-c0f7-480e-a19f-8b77e7285af2" volumeName="kubernetes.io/secret/a9f7db5a-c0f7-480e-a19f-8b77e7285af2-cert" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069143    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3b0f6f7-b58c-49a1-aa66-57cee4ae2998" volumeName="kubernetes.io/configmap/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-serviceca" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069156    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069170    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7" volumeName="kubernetes.io/configmap/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-iptables-alerter-script" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069183    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5806796f-e98d-4eb1-b24c-d4c0cfc057f4" volumeName="kubernetes.io/secret/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-metrics-tls" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069197    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069209    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-web-config" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069223    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-grpc-tls" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069235    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f0811b4-9b91-454b-b715-6aaf519b5e92" volumeName="kubernetes.io/secret/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-tls" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069247    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc926104-1c6c-46f3-9d8a-7f1d8394d713" volumeName="kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-grpc-tls" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069259    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dbf1e42-f480-47a4-8382-6207e2fe7774" volumeName="kubernetes.io/projected/1dbf1e42-f480-47a4-8382-6207e2fe7774-kube-api-access-xn24w" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069274    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5806796f-e98d-4eb1-b24c-d4c0cfc057f4" volumeName="kubernetes.io/configmap/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-config-volume" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069286    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069300    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069331    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069352    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61353437-1060-48e2-89f0-b51beb1fdb60" volumeName="kubernetes.io/empty-dir/61353437-1060-48e2-89f0-b51beb1fdb60-data-volume" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069365    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ea5065d-e1c6-4152-842c-833cd8155772" volumeName="kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069379    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af63014e-5ed6-4213-8838-529749c931cb" volumeName="kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069392    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ece33e-8a7b-418f-aa9e-eaa2fed48273" volumeName="kubernetes.io/empty-dir/13ece33e-8a7b-418f-aa9e-eaa2fed48273-tmp-dir" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069407    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-metrics-client-certs" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069419    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c83af76-9520-4c3d-8aec-3590c97abaf7" volumeName="kubernetes.io/configmap/9c83af76-9520-4c3d-8aec-3590c97abaf7-metrics-client-ca" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069434    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a00bade6-34ba-4d87-a740-9727b8372650" volumeName="kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069450    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0652c5d-5440-4b54-9220-fbf77c06ec2f" volumeName="kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069464    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc926104-1c6c-46f3-9d8a-7f1d8394d713" volumeName="kubernetes.io/empty-dir/cc926104-1c6c-46f3-9d8a-7f1d8394d713-tmp" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069477    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config-out" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069491    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc926104-1c6c-46f3-9d8a-7f1d8394d713" volumeName="kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069506    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069530    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e20bd59-e885-4662-bb6b-7cd403b9d752" volumeName="kubernetes.io/configmap/1e20bd59-e885-4662-bb6b-7cd403b9d752-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069542    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e20bd59-e885-4662-bb6b-7cd403b9d752" volumeName="kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-secret-metrics-server-tls" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069555    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5806796f-e98d-4eb1-b24c-d4c0cfc057f4" volumeName="kubernetes.io/empty-dir/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-tmp-dir" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069568    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e20bd59-e885-4662-bb6b-7cd403b9d752" volumeName="kubernetes.io/empty-dir/1e20bd59-e885-4662-bb6b-7cd403b9d752-audit-log" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069584    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-kube-api-access-kfqjb" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069597    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61353437-1060-48e2-89f0-b51beb1fdb60" volumeName="kubernetes.io/projected/61353437-1060-48e2-89f0-b51beb1fdb60-kube-api-access-66f87" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069611    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dbf1e42-f480-47a4-8382-6207e2fe7774" volumeName="kubernetes.io/configmap/1dbf1e42-f480-47a4-8382-6207e2fe7774-cni-binary-copy" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069624    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069637    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069650    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="596632c1-fb08-4008-ad12-7d399fa3994b" volumeName="kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-env-overrides" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069668    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="596632c1-fb08-4008-ad12-7d399fa3994b" volumeName="kubernetes.io/secret/596632c1-fb08-4008-ad12-7d399fa3994b-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069682    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9f7db5a-c0f7-480e-a19f-8b77e7285af2" volumeName="kubernetes.io/projected/a9f7db5a-c0f7-480e-a19f-8b77e7285af2-kube-api-access-24wql" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069695    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0652c5d-5440-4b54-9220-fbf77c06ec2f" volumeName="kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cni-binary-copy" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069709    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0652c5d-5440-4b54-9220-fbf77c06ec2f" volumeName="kubernetes.io/projected/c0652c5d-5440-4b54-9220-fbf77c06ec2f-kube-api-access-8qrc5" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069722    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="596632c1-fb08-4008-ad12-7d399fa3994b" volumeName="kubernetes.io/projected/596632c1-fb08-4008-ad12-7d399fa3994b-kube-api-access-svwtn" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069734    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc926104-1c6c-46f3-9d8a-7f1d8394d713" volumeName="kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069759    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc926104-1c6c-46f3-9d8a-7f1d8394d713" volumeName="kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069773    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13d056b1-85f8-48eb-b9bf-aeb4b98a8170" volumeName="kubernetes.io/configmap/13d056b1-85f8-48eb-b9bf-aeb4b98a8170-konnectivity-ca" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069785    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="596632c1-fb08-4008-ad12-7d399fa3994b" volumeName="kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069824    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc926104-1c6c-46f3-9d8a-7f1d8394d713" volumeName="kubernetes.io/projected/cc926104-1c6c-46f3-9d8a-7f1d8394d713-kube-api-access-pfqzs" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069849    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13d056b1-85f8-48eb-b9bf-aeb4b98a8170" volumeName="kubernetes.io/secret/13d056b1-85f8-48eb-b9bf-aeb4b98a8170-agent-certs" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069862    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" volumeName="kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-tls-assets" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069876    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c44c5cc-7fb0-40c7-970b-add5c87c0770" volumeName="kubernetes.io/projected/1c44c5cc-7fb0-40c7-970b-add5c87c0770-kube-api-access-wwxn6" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069889    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e20bd59-e885-4662-bb6b-7cd403b9d752" volumeName="kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-client-ca-bundle" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069902    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61353437-1060-48e2-89f0-b51beb1fdb60" volumeName="kubernetes.io/configmap/61353437-1060-48e2-89f0-b51beb1fdb60-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069916    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c83af76-9520-4c3d-8aec-3590c97abaf7" volumeName="kubernetes.io/configmap/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069929    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c83af76-9520-4c3d-8aec-3590c97abaf7" volumeName="kubernetes.io/empty-dir/9c83af76-9520-4c3d-8aec-3590c97abaf7-volume-directive-shadow" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069941    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c83af76-9520-4c3d-8aec-3590c97abaf7" volumeName="kubernetes.io/secret/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069966    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f0811b4-9b91-454b-b715-6aaf519b5e92" volumeName="kubernetes.io/empty-dir/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-textfile" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069980    6506 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c83af76-9520-4c3d-8aec-3590c97abaf7" volumeName="kubernetes.io/projected/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-api-access-q5r8x" seLinuxMountContext=""
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.069992    6506 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 16:03:06.072742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.070000    6506 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 16:03:06.076693 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.075694    6506 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:03:06.076693 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.075854    6506 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:03:06.078669 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.077548    6506 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:03:06.079698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.079677    6506 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 16:03:06.094126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.093974    6506 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 16:03:06.095635 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.095617    6506 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 16:03:06.095635 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.095638    6506 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 16:03:06.095811 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.095657    6506 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 16:03:06.095811 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:06.095696    6506 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 16:03:06.099979 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.099959    6506 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:03:06.122235 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.122197    6506 generic.go:356] "Generic (PLEG): container finished" podID="56a59496f84ad599eee942e332b42fd6" containerID="6625f2354d9716f3a80855252f360d1b293a181169975be43f7f6c278e27c089" exitCode=0
Nov 03 16:03:06.127098 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.127066    6506 generic.go:356] "Generic (PLEG): container finished" podID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerID="916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9" exitCode=0
Nov 03 16:03:06.136813 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.136767    6506 generic.go:356] "Generic (PLEG): container finished" podID="9f0811b4-9b91-454b-b715-6aaf519b5e92" containerID="61e8aa3d32e461ba292b5f720298e82d4164016bdf87189fb4b6e08d39bbd84a" exitCode=0
Nov 03 16:03:06.143669 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.143644    6506 generic.go:356] "Generic (PLEG): container finished" podID="c0652c5d-5440-4b54-9220-fbf77c06ec2f" containerID="a0a69fe79ddc49c8aa9e945e8486632e5f20bde05a4c326c7f52df4a60033a90" exitCode=0
Nov 03 16:03:06.143760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.143671    6506 generic.go:356] "Generic (PLEG): container finished" podID="c0652c5d-5440-4b54-9220-fbf77c06ec2f" containerID="92d40c6fab91fdaca94f6d2f7bf00c6937da427a991ccd99e9e8dba0737b160d" exitCode=0
Nov 03 16:03:06.143760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.143681    6506 generic.go:356] "Generic (PLEG): container finished" podID="c0652c5d-5440-4b54-9220-fbf77c06ec2f" containerID="8bfa8eb45fbe503006ce52cfc6604f851bb4ad45c0fe30a7c2de3e68a10120b2" exitCode=0
Nov 03 16:03:06.143760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.143691    6506 generic.go:356] "Generic (PLEG): container finished" podID="c0652c5d-5440-4b54-9220-fbf77c06ec2f" containerID="ca19d25eab2c1acbbf021e4f363d909352d953e649036fa2aecfc6ade25b27fd" exitCode=0
Nov 03 16:03:06.143760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.143699    6506 generic.go:356] "Generic (PLEG): container finished" podID="c0652c5d-5440-4b54-9220-fbf77c06ec2f" containerID="001e57073919f5d0ec1cf540e3ad856fcaf84e42fb0f1b76e6d253006bc82b60" exitCode=0
Nov 03 16:03:06.143760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.143707    6506 generic.go:356] "Generic (PLEG): container finished" podID="c0652c5d-5440-4b54-9220-fbf77c06ec2f" containerID="03d464390d235d6ca924e4ea4ac28db3de756236b7c06f91c615eb5dd8bbe587" exitCode=0
Nov 03 16:03:06.173174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.173156    6506 manager.go:324] Recovery completed
Nov 03 16:03:06.195661 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.195610    6506 cpu_manager.go:226] "Starting" policy="none"
Nov 03 16:03:06.195661 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.195624    6506 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 16:03:06.195661 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.195641    6506 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 16:03:06.195826 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:06.195730    6506 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 16:03:06.195826 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.195759    6506 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 16:03:06.195826 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.195769    6506 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 16:03:06.195826 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.195784    6506 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 16:03:06.195826 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.195790    6506 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 16:03:06.195826 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.195810    6506 policy_none.go:50] "Start"
Nov 03 16:03:06.195826 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.195819    6506 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 16:03:06.195826 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.195828    6506 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 16:03:06.196043 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.195947    6506 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 16:03:06.196043 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.195954    6506 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 16:03:06.196043 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.195960    6506 policy_none.go:44] "Start"
Nov 03 16:03:06.200330 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.200293    6506 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 16:03:06.202836 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.202822    6506 manager.go:341] "Starting Device Plugin manager"
Nov 03 16:03:06.203325 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:06.202853    6506 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 16:03:06.203362 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.203329    6506 server.go:87] "Starting device plugin registration server"
Nov 03 16:03:06.203580 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.203571    6506 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 16:03:06.203610 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.203582    6506 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 16:03:06.203716 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.203699    6506 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 16:03:06.203837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.203788    6506 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 16:03:06.203933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.203843    6506 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 16:03:06.203933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.203848    6506 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 16:03:06.204052 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.203890    6506 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:03:06.203831181Z","UUID":"2b4ca03b-50c1-4ac3-92e2-33e13b314f16","Handler":null,"Name":"","Endpoint":""}
Nov 03 16:03:06.205333 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.205309    6506 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 16:03:06.205333 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.205332    6506 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 16:03:06.205536 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:06.205511    6506 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 16:03:06.304208 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.304177    6506 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:03:06.306367 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.306347    6506 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:03:06.306491 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.306377    6506 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:03:06.306491 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.306387    6506 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-159.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:03:06.306491 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.306426    6506 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.320353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.320333    6506 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.320429 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.320389    6506 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.396856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.396810    6506 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal"]
Nov 03 16:03:06.397502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.397460    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal" event={"ID":"56a59496f84ad599eee942e332b42fd6","Type":"ContainerStarted","Data":"341b5de13274a7f8c536949691d893d4f0bfe163e7375abc62e87514476b0331"}
Nov 03 16:03:06.397551 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.397515    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal" event={"ID":"56a59496f84ad599eee942e332b42fd6","Type":"ContainerDied","Data":"6625f2354d9716f3a80855252f360d1b293a181169975be43f7f6c278e27c089"}
Nov 03 16:03:06.397551 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.397532    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal" event={"ID":"56a59496f84ad599eee942e332b42fd6","Type":"ContainerStarted","Data":"a1fe67ccc0ffbc19e2e96cbc485098416398466cf149c1884a554c830799fa00"}
Nov 03 16:03:06.397651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.397565    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal" event={"ID":"f898674fe0c94d2ff8a087a4cd76c349","Type":"ContainerStarted","Data":"c6eb742b705ff79bd912977157c9c7eec84a4571818b205700965f88ba846e78"}
Nov 03 16:03:06.397651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.397578    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal" event={"ID":"f898674fe0c94d2ff8a087a4cd76c349","Type":"ContainerStarted","Data":"d0fb0377af2f674020219c26fa4a1f42c72cea83140ef2909ceca1f643b17f94"}
Nov 03 16:03:06.397651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.397493    6506 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.397787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.397701    6506 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.420018 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.419998    6506 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:03:06.420094 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:06.420037    6506 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.420672 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.420657    6506 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:03:06.420748 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:06.420697    6506 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.481036 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.480975    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56a59496f84ad599eee942e332b42fd6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"56a59496f84ad599eee942e332b42fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.481036 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.481009    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56a59496f84ad599eee942e332b42fd6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"56a59496f84ad599eee942e332b42fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.481036 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.481034    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f898674fe0c94d2ff8a087a4cd76c349-config\") pod \"kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"f898674fe0c94d2ff8a087a4cd76c349\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.581587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.581555    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56a59496f84ad599eee942e332b42fd6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"56a59496f84ad599eee942e332b42fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.581700 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.581591    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f898674fe0c94d2ff8a087a4cd76c349-config\") pod \"kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"f898674fe0c94d2ff8a087a4cd76c349\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.581700 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.581610    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56a59496f84ad599eee942e332b42fd6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"56a59496f84ad599eee942e332b42fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.581700 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.581652    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f898674fe0c94d2ff8a087a4cd76c349-config\") pod \"kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"f898674fe0c94d2ff8a087a4cd76c349\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.581700 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.581661    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56a59496f84ad599eee942e332b42fd6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"56a59496f84ad599eee942e332b42fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:06.581840 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:06.581706    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56a59496f84ad599eee942e332b42fd6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal\" (UID: \"56a59496f84ad599eee942e332b42fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:07.049988 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.049958    6506 apiserver.go:52] "Watching apiserver"
Nov 03 16:03:07.059538 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.059494    6506 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 16:03:07.060107 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.060086    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0","openshift-multus/multus-additional-cni-plugins-vsg95","openshift-multus/multus-n4g6g","kube-system/global-pull-secret-syncer-qmd98","openshift-dns/node-resolver-7f6nb","openshift-dns/dns-default-kgdhx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal","openshift-insights/insights-runtime-extractor-hdzxp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d","openshift-ingress-canary/ingress-canary-8vvpp","kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal","openshift-monitoring/metrics-server-56b6dcb5b8-8jqch","openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq","openshift-multus/network-metrics-daemon-sf4cc","openshift-monitoring/node-exporter-lvq8q","openshift-network-operator/iptables-alerter-5flz8","openshift-image-registry/node-ca-j5r6g","kube-system/konnectivity-agent-hdgx7","openshift-monitoring/thanos-querier-7747889c6f-6psbm","openshift-cluster-node-tuning-operator/tuned-rlgrt","openshift-network-diagnostics/network-check-target-2ht7q","openshift-ovn-kubernetes/ovnkube-node-2vtl4"]
Nov 03 16:03:07.066947 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.066921    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 16:03:07.067089 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.067044    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d4csl\""
Nov 03 16:03:07.067196 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.067179    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 16:03:07.067262 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.067182    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xkdws\""
Nov 03 16:03:07.067468 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.067450    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4r589\""
Nov 03 16:03:07.067676 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.067657    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dchj7\""
Nov 03 16:03:07.067745 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.067696    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qk9pq\""
Nov 03 16:03:07.068478 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.068453    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lgwbg\""
Nov 03 16:03:07.068599 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.068583    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c8h5g\""
Nov 03 16:03:07.068725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.068705    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 16:03:07.068930 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.068913    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 16:03:07.069012 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.068952    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jdvsk\""
Nov 03 16:03:07.069136 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.069115    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c56qh\""
Nov 03 16:03:07.069241 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.069138    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5rhh2\""
Nov 03 16:03:07.069344 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.069310    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 16:03:07.070439 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070417    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 16:03:07.070540 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070419    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 16:03:07.070540 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070469    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 16:03:07.070540 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070504    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 16:03:07.070540 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070521    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 16:03:07.070540 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070526    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 16:03:07.070781 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070762    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 16:03:07.070861 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070809    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:03:07.070861 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070826    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 16:03:07.070861 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070858    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 16:03:07.071009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070862    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 16:03:07.071009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070901    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 16:03:07.071009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070905    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 16:03:07.071009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070859    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 16:03:07.071009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070917    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-drdlv\""
Nov 03 16:03:07.071009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070939    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 16:03:07.071009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.070981    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 16:03:07.071264 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.071018    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 16:03:07.071264 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.071016    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 16:03:07.071264 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.071124    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 16:03:07.071372 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.071347    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 16:03:07.072013 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.071992    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 16:03:07.072174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.072159    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:03:07.072231 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.072160    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 16:03:07.072231 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.072196    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 16:03:07.072308 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.072295    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2h492\""
Nov 03 16:03:07.072346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.072330    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 16:03:07.072393 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.072378    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 16:03:07.073052 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.073037    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 16:03:07.073754 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.073736    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 16:03:07.074339 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.074317    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jk68z\""
Nov 03 16:03:07.076960 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.076942    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 16:03:07.076960 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.076954    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 16:03:07.077057 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.076964    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 16:03:07.077108 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.077056    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 16:03:07.077478 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.077464    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 16:03:07.078196 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.078183    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 16:03:07.078253 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.078232    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8szjr\""
Nov 03 16:03:07.078324 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.078300    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6bb2h\""
Nov 03 16:03:07.078356 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.078328    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 16:03:07.078461 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.078450    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 16:03:07.078534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.078523    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 16:03:07.079106 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.079094    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-56b2k\""
Nov 03 16:03:07.079249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.079237    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 16:03:07.079278 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.079252    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 16:03:07.079775 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.079762    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 16:03:07.080130 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.080117    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 16:03:07.080245 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.080233    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 16:03:07.080299 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.080237    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 16:03:07.082030 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.082014    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 16:03:07.082426 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.082412    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 16:03:07.082474 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.082424    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-d2hch\""
Nov 03 16:03:07.082506 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.082483    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 16:03:07.082552 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.082542    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xrtx4\""
Nov 03 16:03:07.082613 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.082599    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b791nd51864r5\""
Nov 03 16:03:07.082763 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.082746    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 16:03:07.083401 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.083385    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 16:03:07.083455 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.083404    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 16:03:07.083455 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.083426    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 16:03:07.083617 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.083603    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-socket-dir-parent\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.083647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.083626    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.083647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.083641    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-lib-modules\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.083760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.083657    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-cni-bin\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.083760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.083677    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-k8s-cni-cncf-io\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.083760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.083697    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-netns\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.083760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.083723    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.083760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.083748    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-cni-netd\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.083975 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.083760    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 16:03:07.083975 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.083834    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13ece33e-8a7b-418f-aa9e-eaa2fed48273-hosts-file\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:03:07.083975 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.083886    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-kubernetes\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.083975 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.083966    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-systemd\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.084103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084038    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qrc5\" (UniqueName: \"kubernetes.io/projected/c0652c5d-5440-4b54-9220-fbf77c06ec2f-kube-api-access-8qrc5\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.084103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084076    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs5sr\" (UniqueName: \"kubernetes.io/projected/13ece33e-8a7b-418f-aa9e-eaa2fed48273-kube-api-access-rs5sr\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:03:07.084103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084096    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-tuned\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.084215 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084112    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cni-binary-copy\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.084215 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084127    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4774\" (UniqueName: \"kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774\") pod \"network-check-target-2ht7q\" (UID: \"af63014e-5ed6-4213-8838-529749c931cb\") " pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:03:07.084215 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084155    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-cni-multus\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.084346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084174    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-multus-certs\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.084346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084244    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-tuned\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.084346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084250    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-var-lib-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.084346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084303    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-ovn\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.084483 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084341    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysctl-d\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.084483 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084373    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:03:07.084483 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084388    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cni-binary-copy\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.084596 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084475    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-system-cni-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.084596 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084511    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-conf-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.084596 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084532    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-slash\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.084596 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084559    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00bade6-34ba-4d87-a740-9727b8372650-metrics-certs\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:03:07.084596 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084571    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-log-socket\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.084831 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084601    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-run\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.084831 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084630    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-cnibin\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.084831 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084650    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-hostroot\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.084831 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084686    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-kubelet\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.084831 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084718    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svwtn\" (UniqueName: \"kubernetes.io/projected/596632c1-fb08-4008-ad12-7d399fa3994b-kube-api-access-svwtn\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.084831 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084743    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/596632c1-fb08-4008-ad12-7d399fa3994b-ovn-node-metrics-cert\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.084831 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084817    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-etc-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.085145 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084899    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.085145 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084932    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-modprobe-d\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.085145 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084955    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-sys\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.085145 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.084994    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/596632c1-fb08-4008-ad12-7d399fa3994b-ovn-node-metrics-cert\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.085145 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085000    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-var-lib-kubelet\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.085145 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085047    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.085145 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085142    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.085465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085163    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:03:07.085465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085178    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-env-overrides\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.085465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085198    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cnibin\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.085465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085228    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.085465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085263    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-daemon-config\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.085465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085287    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7ea5065d-e1c6-4152-842c-833cd8155772-kubelet-config\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:03:07.085465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085314    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ece33e-8a7b-418f-aa9e-eaa2fed48273-tmp-dir\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:03:07.085465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085337    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c44c5cc-7fb0-40c7-970b-add5c87c0770-tmp\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.085465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085382    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7ea5065d-e1c6-4152-842c-833cd8155772-dbus\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:03:07.085465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085402    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.085465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085414    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c44c5cc-7fb0-40c7-970b-add5c87c0770-tmp\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.085465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085418    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn24w\" (UniqueName: \"kubernetes.io/projected/1dbf1e42-f480-47a4-8382-6207e2fe7774-kube-api-access-xn24w\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.085465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085456    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-cni-bin\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085482    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-daemon-config\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085501    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-env-overrides\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085512    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-ovnkube-config\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085515    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ea5065d-e1c6-4152-842c-833cd8155772-original-pull-secret\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085592    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085602    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-systemd-units\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085630    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-system-cni-dir\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085652    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-host\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085700    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwxn6\" (UniqueName: \"kubernetes.io/projected/1c44c5cc-7fb0-40c7-970b-add5c87c0770-kube-api-access-wwxn6\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085757    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hsrw\" (UniqueName: \"kubernetes.io/projected/a00bade6-34ba-4d87-a740-9727b8372650-kube-api-access-9hsrw\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085772    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ece33e-8a7b-418f-aa9e-eaa2fed48273-tmp-dir\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085782    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dbf1e42-f480-47a4-8382-6207e2fe7774-cni-binary-copy\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085815    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-ovnkube-config\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085822    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-run-netns\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085905    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysconfig\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085931    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-systemd\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085953    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-etc-kubernetes\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.085982    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-ovnkube-script-lib\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086008    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-kubelet\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086021    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086029    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-node-log\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086052    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysctl-conf\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086061    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086074    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-cni-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.086088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086098    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 16:03:07.087066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086019    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dbf1e42-f480-47a4-8382-6207e2fe7774-cni-binary-copy\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.087066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086121    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 16:03:07.087066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086054    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 16:03:07.087066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086174    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 16:03:07.087066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086179    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 16:03:07.087066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086202    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 16:03:07.087066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086226    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 16:03:07.087066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086625    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 16:03:07.087066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086701    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/596632c1-fb08-4008-ad12-7d399fa3994b-ovnkube-script-lib\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.087066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086710    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 16:03:07.087066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086854    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 16:03:07.087066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.086934    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5jf2w\""
Nov 03 16:03:07.087066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.087029    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8qdq8\""
Nov 03 16:03:07.087562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.087108    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 16:03:07.087562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.087286    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 16:03:07.087562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.087327    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ei2tuiaibn95i\""
Nov 03 16:03:07.087562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.087532    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 16:03:07.087826 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.087784    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 16:03:07.088044 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.088026    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-duq0m103irmgu\""
Nov 03 16:03:07.096129 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.096112    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qrc5\" (UniqueName: \"kubernetes.io/projected/c0652c5d-5440-4b54-9220-fbf77c06ec2f-kube-api-access-8qrc5\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.096349 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.096334    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs5sr\" (UniqueName: \"kubernetes.io/projected/13ece33e-8a7b-418f-aa9e-eaa2fed48273-kube-api-access-rs5sr\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:03:07.098400 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.098381    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn24w\" (UniqueName: \"kubernetes.io/projected/1dbf1e42-f480-47a4-8382-6207e2fe7774-kube-api-access-xn24w\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.098502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.098433    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwxn6\" (UniqueName: \"kubernetes.io/projected/1c44c5cc-7fb0-40c7-970b-add5c87c0770-kube-api-access-wwxn6\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.098702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.098685    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hsrw\" (UniqueName: \"kubernetes.io/projected/a00bade6-34ba-4d87-a740-9727b8372650-kube-api-access-9hsrw\") pod \"network-metrics-daemon-sf4cc\" (UID: \"a00bade6-34ba-4d87-a740-9727b8372650\") " pod="openshift-multus/network-metrics-daemon-sf4cc"
Nov 03 16:03:07.098927 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.098910    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4774\" (UniqueName: \"kubernetes.io/projected/af63014e-5ed6-4213-8838-529749c931cb-kube-api-access-r4774\") pod \"network-check-target-2ht7q\" (UID: \"af63014e-5ed6-4213-8838-529749c931cb\") " pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:03:07.099111 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.099094    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svwtn\" (UniqueName: \"kubernetes.io/projected/596632c1-fb08-4008-ad12-7d399fa3994b-kube-api-access-svwtn\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.152321 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.152300    6506 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:07.152457 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.152443    6506 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:07.163656 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.163633    6506 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:03:07.163748 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:07.163668    6506 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:07.164780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.164760    6506 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 16:03:07.170397 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.170382    6506 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:03:07.170488 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:07.170429    6506 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-159.us-east-2.compute.internal"
Nov 03 16:03:07.187181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187156    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-systemd-units\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.187268 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187186    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-system-cni-dir\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.187268 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187210    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-host\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.187268 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187233    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:03:07.187268 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187259    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-run-netns\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.187418 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187270    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-systemd-units\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.187418 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187276    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-system-cni-dir\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.187418 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187305    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-host\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.187418 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187321    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-run-netns\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.187418 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187332    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysconfig\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.187418 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187281    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysconfig\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.187418 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187361    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-systemd\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.187418 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187385    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-tls\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.187418 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187411    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-etc-kubernetes\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.187418 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187421    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-systemd\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187436    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187450    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-etc-kubernetes\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187462    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-748gh\" (UniqueName: \"kubernetes.io/projected/9f0811b4-9b91-454b-b715-6aaf519b5e92-kube-api-access-748gh\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187489    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-node-log\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187512    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysctl-conf\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187520    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187536    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-host-slash\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187559    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187554    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-node-log\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187617    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187657    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-tls\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187671    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187659    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysctl-conf\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187709    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187713    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-lib-modules\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187731    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-textfile\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187730    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187775    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-textfile\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187779    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-cni-bin\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187830    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-cni-bin\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187839    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-k8s-cni-cncf-io\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187868    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-netns\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.187879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187890    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187867    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-lib-modules\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187907    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-cni-netd\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187928    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187931    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13ece33e-8a7b-418f-aa9e-eaa2fed48273-hosts-file\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187949    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-k8s-cni-cncf-io\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187956    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-kubernetes\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187932    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-netns\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187980    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13ece33e-8a7b-418f-aa9e-eaa2fed48273-hosts-file\") pod \"node-resolver-7f6nb\" (UID: \"13ece33e-8a7b-418f-aa9e-eaa2fed48273\") " pod="openshift-dns/node-resolver-7f6nb"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187953    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-cni-netd\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.187988    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw9h4\" (UniqueName: \"kubernetes.io/projected/1e20bd59-e885-4662-bb6b-7cd403b9d752-kube-api-access-fw9h4\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188024    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-systemd\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188008    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-kubernetes\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188058    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-systemd\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188056    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61353437-1060-48e2-89f0-b51beb1fdb60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188083    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188097    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188111    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188135    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-sys-fs\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188159    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-tmp-dir\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188179    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188205    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188243    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc926104-1c6c-46f3-9d8a-7f1d8394d713-tmp\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188250    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61353437-1060-48e2-89f0-b51beb1fdb60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188246    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-tmp-dir\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188269    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-multus-certs\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188268    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188320    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-run-multus-certs\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188301    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-secret-metrics-server-client-certs\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188353    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c83af76-9520-4c3d-8aec-3590c97abaf7-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188361    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc926104-1c6c-46f3-9d8a-7f1d8394d713-tmp\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188378    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-system-cni-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188378    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188402    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-conf-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188418    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188427    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-log-socket\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188460    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-run\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188465    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-system-cni-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188470    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-log-socket\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188492    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-conf-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188497    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66f87\" (UniqueName: \"kubernetes.io/projected/61353437-1060-48e2-89f0-b51beb1fdb60-kube-api-access-66f87\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188514    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-run\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188516    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-secret-metrics-server-client-certs\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188529    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c83af76-9520-4c3d-8aec-3590c97abaf7-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188539    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188564    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-web-config\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188588    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188615    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-cnibin\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.188626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188643    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188676    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188706    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188727    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188723    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-cnibin\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188746    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188755    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-serviceca\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188785    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188837    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9jz2\" (UniqueName: \"kubernetes.io/projected/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-kube-api-access-x9jz2\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188862    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cnibin\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188875    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188892    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61353437-1060-48e2-89f0-b51beb1fdb60-crio-socket\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188918    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-sys\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188941    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-iptables-alerter-script\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188950    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188955    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188967    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5r8x\" (UniqueName: \"kubernetes.io/projected/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-api-access-q5r8x\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189010    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0652c5d-5440-4b54-9220-fbf77c06ec2f-cnibin\") pod \"multus-additional-cni-plugins-vsg95\" (UID: \"c0652c5d-5440-4b54-9220-fbf77c06ec2f\") " pod="openshift-multus/multus-additional-cni-plugins-vsg95"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189024    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.188748    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-web-config\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189162    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-iptables-alerter-script\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189154    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-grpc-tls\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189305    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189329    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189344    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9f7db5a-c0f7-480e-a19f-8b77e7285af2-cert\") pod \"ingress-canary-8vvpp\" (UID: \"a9f7db5a-c0f7-480e-a19f-8b77e7285af2\") " pod="openshift-ingress-canary/ingress-canary-8vvpp"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189359    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189374    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfqjb\" (UniqueName: \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-kube-api-access-kfqjb\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189391    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24wql\" (UniqueName: \"kubernetes.io/projected/a9f7db5a-c0f7-480e-a19f-8b77e7285af2-kube-api-access-24wql\") pod \"ingress-canary-8vvpp\" (UID: \"a9f7db5a-c0f7-480e-a19f-8b77e7285af2\") " pod="openshift-ingress-canary/ingress-canary-8vvpp"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189406    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-grpc-tls\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189414    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-client-ca-bundle\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189499    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-serviceca\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189513    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-root\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189543    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfqzs\" (UniqueName: \"kubernetes.io/projected/cc926104-1c6c-46f3-9d8a-7f1d8394d713-kube-api-access-pfqzs\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189579    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9f7db5a-c0f7-480e-a19f-8b77e7285af2-cert\") pod \"ingress-canary-8vvpp\" (UID: \"a9f7db5a-c0f7-480e-a19f-8b77e7285af2\") " pod="openshift-ingress-canary/ingress-canary-8vvpp"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189590    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189587    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189628    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-client-ca-bundle\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189624    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189661    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9c83af76-9520-4c3d-8aec-3590c97abaf7-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189684    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkcbp\" (UniqueName: \"kubernetes.io/projected/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-kube-api-access-qkcbp\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189721    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-tls\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189746    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-registration-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189774    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-kubelet\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189834    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9c83af76-9520-4c3d-8aec-3590c97abaf7-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189838    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189844    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f0811b4-9b91-454b-b715-6aaf519b5e92-metrics-client-ca\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189899    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-kubelet\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.189940    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-cni-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190017    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-tls\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190042    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-socket-dir-parent\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190061    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190066    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-cni-dir\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190077    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e20bd59-e885-4662-bb6b-7cd403b9d752-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190086    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f0811b4-9b91-454b-b715-6aaf519b5e92-metrics-client-ca\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190108    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-multus-socket-dir-parent\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190113    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190203    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190233    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190234    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e20bd59-e885-4662-bb6b-7cd403b9d752-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190266    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1e20bd59-e885-4662-bb6b-7cd403b9d752-audit-log\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190294    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-cni-multus\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190328    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-var-lib-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190334    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190349    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190350    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-ovn\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190384    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-run-ovn\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190387    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-var-lib-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190395    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1e20bd59-e885-4662-bb6b-7cd403b9d752-audit-log\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190393    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysctl-d\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190440    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190456    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-slash\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190405    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190472    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190474    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-sysctl-d\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190501    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-slash\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190501    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-device-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190545    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-host-var-lib-cni-multus\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190544    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-hostroot\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190575    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-kubelet\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.190581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190611    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1dbf1e42-f480-47a4-8382-6207e2fe7774-hostroot\") pod \"multus-n4g6g\" (UID: \"1dbf1e42-f480-47a4-8382-6207e2fe7774\") " pod="openshift-multus/multus-n4g6g"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190629    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190671    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config-out\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190680    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-kubelet\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190722    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krj9t\" (UniqueName: \"kubernetes.io/projected/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-kube-api-access-krj9t\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190747    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config-out\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190780    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190828    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190851    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-etc-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190866    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190881    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-modprobe-d\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190902    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-sys\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190913    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-etc-openvswitch\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190926    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-var-lib-kubelet\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190929    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190960    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61353437-1060-48e2-89f0-b51beb1fdb60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190979    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-var-lib-kubelet\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190987    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-wtmp\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.190979    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-sys\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191011    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c44c5cc-7fb0-40c7-970b-add5c87c0770-etc-modprobe-d\") pod \"tuned-rlgrt\" (UID: \"1c44c5cc-7fb0-40c7-970b-add5c87c0770\") " pod="openshift-cluster-node-tuning-operator/tuned-rlgrt"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191024    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191041    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1e20bd59-e885-4662-bb6b-7cd403b9d752-metrics-server-audit-profiles\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191056    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc926104-1c6c-46f3-9d8a-7f1d8394d713-metrics-client-ca\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191057    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191077    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13d056b1-85f8-48eb-b9bf-aeb4b98a8170-konnectivity-ca\") pod \"konnectivity-agent-hdgx7\" (UID: \"13d056b1-85f8-48eb-b9bf-aeb4b98a8170\") " pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191102    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191129    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-socket-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191196    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191201    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61353437-1060-48e2-89f0-b51beb1fdb60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191224    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-metrics-tls\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191243    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc926104-1c6c-46f3-9d8a-7f1d8394d713-metrics-client-ca\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191260    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191266    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-secret-metrics-server-tls\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191295    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1e20bd59-e885-4662-bb6b-7cd403b9d752-metrics-server-audit-profiles\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191298    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13d056b1-85f8-48eb-b9bf-aeb4b98a8170-konnectivity-ca\") pod \"konnectivity-agent-hdgx7\" (UID: \"13d056b1-85f8-48eb-b9bf-aeb4b98a8170\") " pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191376    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-host\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191430    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191445    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-metrics-tls\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191453    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61353437-1060-48e2-89f0-b51beb1fdb60-data-volume\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191483    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191490    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61353437-1060-48e2-89f0-b51beb1fdb60-data-volume\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191512    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191540    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7s62\" (UniqueName: \"kubernetes.io/projected/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-kube-api-access-v7s62\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191557    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191566    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-config-volume\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191597    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7ea5065d-e1c6-4152-842c-833cd8155772-kubelet-config\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191621    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7ea5065d-e1c6-4152-842c-833cd8155772-dbus\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191540    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1e20bd59-e885-4662-bb6b-7cd403b9d752-secret-metrics-server-tls\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191638    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191645    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-cni-bin\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191675    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13d056b1-85f8-48eb-b9bf-aeb4b98a8170-agent-certs\") pod \"konnectivity-agent-hdgx7\" (UID: \"13d056b1-85f8-48eb-b9bf-aeb4b98a8170\") " pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191697    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7ea5065d-e1c6-4152-842c-833cd8155772-kubelet-config\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191707    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7ea5065d-e1c6-4152-842c-833cd8155772-dbus\") pod \"global-pull-secret-syncer-qmd98\" (UID: \"7ea5065d-e1c6-4152-842c-833cd8155772\") " pod="kube-system/global-pull-secret-syncer-qmd98"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191726    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc926104-1c6c-46f3-9d8a-7f1d8394d713-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191737    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/596632c1-fb08-4008-ad12-7d399fa3994b-host-cni-bin\") pod \"ovnkube-node-2vtl4\" (UID: \"596632c1-fb08-4008-ad12-7d399fa3994b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191751    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-config-volume\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:03:07.193213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.191884    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13d056b1-85f8-48eb-b9bf-aeb4b98a8170-agent-certs\") pod \"konnectivity-agent-hdgx7\" (UID: \"13d056b1-85f8-48eb-b9bf-aeb4b98a8170\") " pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:03:07.198873 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.198851    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-748gh\" (UniqueName: \"kubernetes.io/projected/9f0811b4-9b91-454b-b715-6aaf519b5e92-kube-api-access-748gh\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.205092 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.205060    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9jz2\" (UniqueName: \"kubernetes.io/projected/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-kube-api-access-x9jz2\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.205290 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.205274    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5r8x\" (UniqueName: \"kubernetes.io/projected/9c83af76-9520-4c3d-8aec-3590c97abaf7-kube-api-access-q5r8x\") pod \"kube-state-metrics-5dc8d7d89b-w7fdq\" (UID: \"9c83af76-9520-4c3d-8aec-3590c97abaf7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7fdq"
Nov 03 16:03:07.205340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.205292    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkcbp\" (UniqueName: \"kubernetes.io/projected/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-kube-api-access-qkcbp\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:03:07.205443 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.205429    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw9h4\" (UniqueName: \"kubernetes.io/projected/1e20bd59-e885-4662-bb6b-7cd403b9d752-kube-api-access-fw9h4\") pod \"metrics-server-56b6dcb5b8-8jqch\" (UID: \"1e20bd59-e885-4662-bb6b-7cd403b9d752\") " pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:03:07.205652 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.205632    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfqjb\" (UniqueName: \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-kube-api-access-kfqjb\") pod \"prometheus-k8s-0\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:07.205697 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.205681    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krj9t\" (UniqueName: \"kubernetes.io/projected/5806796f-e98d-4eb1-b24c-d4c0cfc057f4-kube-api-access-krj9t\") pod \"dns-default-kgdhx\" (UID: \"5806796f-e98d-4eb1-b24c-d4c0cfc057f4\") " pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:03:07.205889 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.205873    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66f87\" (UniqueName: \"kubernetes.io/projected/61353437-1060-48e2-89f0-b51beb1fdb60-kube-api-access-66f87\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:03:07.206349 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.206332    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24wql\" (UniqueName: \"kubernetes.io/projected/a9f7db5a-c0f7-480e-a19f-8b77e7285af2-kube-api-access-24wql\") pod \"ingress-canary-8vvpp\" (UID: \"a9f7db5a-c0f7-480e-a19f-8b77e7285af2\") " pod="openshift-ingress-canary/ingress-canary-8vvpp"
Nov 03 16:03:07.206388 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.206372    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7s62\" (UniqueName: \"kubernetes.io/projected/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-kube-api-access-v7s62\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:03:07.211133 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.211117    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfqzs\" (UniqueName: \"kubernetes.io/projected/cc926104-1c6c-46f3-9d8a-7f1d8394d713-kube-api-access-pfqzs\") pod \"thanos-querier-7747889c6f-6psbm\" (UID: \"cc926104-1c6c-46f3-9d8a-7f1d8394d713\") " pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:07.292335 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292298    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-host-slash\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:03:07.292335 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292336    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-sys-fs\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.292537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292365    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.292537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292403    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7-host-slash\") pod \"iptables-alerter-5flz8\" (UID: \"27e65ea9-6a1d-4473-9c23-a5ecf3d4bfa7\") " pod="openshift-network-operator/iptables-alerter-5flz8"
Nov 03 16:03:07.292537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292425    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.292537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292429    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-sys-fs\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.292537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292466    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61353437-1060-48e2-89f0-b51beb1fdb60-crio-socket\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:03:07.292537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292487    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-sys\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.292537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292513    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-root\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.292537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292524    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61353437-1060-48e2-89f0-b51beb1fdb60-crio-socket\") pod \"insights-runtime-extractor-hdzxp\" (UID: \"61353437-1060-48e2-89f0-b51beb1fdb60\") " pod="openshift-insights/insights-runtime-extractor-hdzxp"
Nov 03 16:03:07.292875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292546    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-registration-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.292875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292549    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-root\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.292875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292554    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-sys\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.292875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292593    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-device-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.292875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292625    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-wtmp\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.292875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292591    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-registration-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.292875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292641    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-device-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.292875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292656    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-socket-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.292875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292681    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.292875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292690    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f0811b4-9b91-454b-b715-6aaf519b5e92-node-exporter-wtmp\") pod \"node-exporter-lvq8q\" (UID: \"9f0811b4-9b91-454b-b715-6aaf519b5e92\") " pod="openshift-monitoring/node-exporter-lvq8q"
Nov 03 16:03:07.292875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292695    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-host\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:03:07.292875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292727    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3b0f6f7-b58c-49a1-aa66-57cee4ae2998-host\") pod \"node-ca-j5r6g\" (UID: \"e3b0f6f7-b58c-49a1-aa66-57cee4ae2998\") " pod="openshift-image-registry/node-ca-j5r6g"
Nov 03 16:03:07.292875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292752    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-socket-dir\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.292875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.292788    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baadfd63-5ceb-48e0-a5c4-c3d2138e18c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-j8z7d\" (UID: \"baadfd63-5ceb-48e0-a5c4-c3d2138e18c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8z7d"
Nov 03 16:03:07.965627 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:07.965592    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:08.131911 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:08.131880    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:08.352522 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:08.352485    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:08.546493 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:08.546462    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:08.902271 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:08.902241    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:09.033200 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:09.033174    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kgdhx"
Nov 03 16:03:09.058948 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:09.058923    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:09.227861 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:09.227759    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:09.696177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:09.696152    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:09.696412 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:09.696398    6506 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 16:03:11.359236 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:11.359203    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:03:11.490857 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:11.490832    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7747889c6f-6psbm"
Nov 03 16:03:11.978900 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:11.978865    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:12.130771 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:12.130712    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:13.568887 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:13.568857    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-56b6dcb5b8-8jqch"
Nov 03 16:03:14.290029 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:14.289993    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:14.436379 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:14.436345    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:16.355631 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:16.355560    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vtl4"
Nov 03 16:03:17.094685 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:17.094652    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:17.274599 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:17.274568    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:17.693788 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:17.693756    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:21.583565 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:21.583534    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hdgx7"
Nov 03 16:03:22.849577 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:22.849547    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 16:03:22.850119 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:22.850077    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="config-reloader" containerID="cri-o://bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89" gracePeriod=600
Nov 03 16:03:22.850119 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:22.850075    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b" gracePeriod=600
Nov 03 16:03:22.850332 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:22.850130    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="kube-rbac-proxy-web" containerID="cri-o://fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7" gracePeriod=600
Nov 03 16:03:22.850332 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:22.850129    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="kube-rbac-proxy" containerID="cri-o://57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f" gracePeriod=600
Nov 03 16:03:22.850332 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:22.850149    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="thanos-sidecar" containerID="cri-o://70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a" gracePeriod=600
Nov 03 16:03:22.850332 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:22.850187    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="prometheus" containerID="cri-o://8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069" gracePeriod=600
Nov 03 16:03:23.086608 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.086585    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.206295 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206206    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-metrics-client-certs\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206295 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206256    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-1\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206295 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206285    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config-out\" (UniqueName: \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config-out\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206316    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-grpc-tls\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206341    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-0\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206378    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-db\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206406    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-web-config\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-web-config\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206438    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-kube-rbac-proxy\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206476    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206505    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206536    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-trusted-ca-bundle\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206991 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206565    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-tls-assets\" (UniqueName: \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-tls-assets\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206991 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206603    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-tls\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206991 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206633    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206991 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206664    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-metrics-client-ca\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206991 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206699    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-serving-certs-ca-bundle\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206991 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206730    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-kube-api-access-kfqjb\" (UniqueName: \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-kube-api-access-kfqjb\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206991 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206731    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-1" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:03:23.206991 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206768    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-kubelet-serving-ca-bundle\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206991 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206828    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-thanos-prometheus-http-client-file\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.206991 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.206876    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-2\") pod \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\" (UID: \"587c9773-ff2c-4c1d-a4ed-f51fc12ee704\") "
Nov 03 16:03:23.207651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.207082    6506 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.207651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.207375    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-2" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:03:23.207757 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.207713    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-trusted-ca-bundle" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:03:23.208093 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.208060    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-db" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:03:23.208216 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.208068    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-0" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:03:23.208983 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.208955    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-serving-certs-ca-bundle" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:03:23.209578 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.209552    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-kubelet-serving-ca-bundle" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:03:23.210322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.210254    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-grpc-tls" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:03:23.210647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.210617    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-metrics-client-ca" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:03:23.211065 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.211023    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config-out" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:03:23.211924 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.211890    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-metrics-client-certs" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:03:23.212051 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212026    6506 generic.go:356] "Generic (PLEG): container finished" podID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerID="5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b" exitCode=0
Nov 03 16:03:23.212051 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212049    6506 generic.go:356] "Generic (PLEG): container finished" podID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerID="57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f" exitCode=0
Nov 03 16:03:23.212051 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212055    6506 generic.go:356] "Generic (PLEG): container finished" podID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerID="fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7" exitCode=0
Nov 03 16:03:23.212274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212060    6506 generic.go:356] "Generic (PLEG): container finished" podID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerID="70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a" exitCode=0
Nov 03 16:03:23.212274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212065    6506 generic.go:356] "Generic (PLEG): container finished" podID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerID="bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89" exitCode=0
Nov 03 16:03:23.212274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212072    6506 generic.go:356] "Generic (PLEG): container finished" podID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerID="8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069" exitCode=0
Nov 03 16:03:23.212274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212128    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"587c9773-ff2c-4c1d-a4ed-f51fc12ee704","Type":"ContainerDied","Data":"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"}
Nov 03 16:03:23.212274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212139    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.212274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212193    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"587c9773-ff2c-4c1d-a4ed-f51fc12ee704","Type":"ContainerDied","Data":"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"}
Nov 03 16:03:23.212274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212221    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"587c9773-ff2c-4c1d-a4ed-f51fc12ee704","Type":"ContainerDied","Data":"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"}
Nov 03 16:03:23.212274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212251    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"587c9773-ff2c-4c1d-a4ed-f51fc12ee704","Type":"ContainerDied","Data":"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"}
Nov 03 16:03:23.212274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212276    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"587c9773-ff2c-4c1d-a4ed-f51fc12ee704","Type":"ContainerDied","Data":"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"}
Nov 03 16:03:23.212616 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212299    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"587c9773-ff2c-4c1d-a4ed-f51fc12ee704","Type":"ContainerDied","Data":"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"}
Nov 03 16:03:23.212616 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212331    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"587c9773-ff2c-4c1d-a4ed-f51fc12ee704","Type":"ContainerDied","Data":"3c1e5ce477e74f85116c5361dffe10a499ce6966005b6620de86c37ed3734ae9"}
Nov 03 16:03:23.212616 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212366    6506 scope.go:122] "RemoveContainer" containerID="5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"
Nov 03 16:03:23.212819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212776    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-kube-rbac-proxy-web" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:03:23.212819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212781    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-tls-assets" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:03:23.212972 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.212887    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-tls" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:03:23.213048 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.213027    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-thanos-sidecar-tls" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:03:23.213130 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.213106    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-kube-rbac-proxy" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:03:23.213300 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.213280    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-thanos-prometheus-http-client-file" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:03:23.213350 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.213307    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-kube-api-access-kfqjb" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "kube-api-access-kfqjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:03:23.213641 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.213579    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:03:23.220994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.220977    6506 scope.go:122] "RemoveContainer" containerID="57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"
Nov 03 16:03:23.221534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.221513    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-web-config" pod "587c9773-ff2c-4c1d-a4ed-f51fc12ee704" (UID: "587c9773-ff2c-4c1d-a4ed-f51fc12ee704"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:03:23.227550 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.227536    6506 scope.go:122] "RemoveContainer" containerID="fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"
Nov 03 16:03:23.233909 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.233896    6506 scope.go:122] "RemoveContainer" containerID="70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"
Nov 03 16:03:23.240020 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.239993    6506 scope.go:122] "RemoveContainer" containerID="bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"
Nov 03 16:03:23.246162 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.246145    6506 scope.go:122] "RemoveContainer" containerID="8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"
Nov 03 16:03:23.252704 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.252688    6506 scope.go:122] "RemoveContainer" containerID="916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9"
Nov 03 16:03:23.258969 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.258956    6506 scope.go:122] "RemoveContainer" containerID="5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"
Nov 03 16:03:23.259204 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:23.259187    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b\": container with ID starting with 5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b not found: ID does not exist" containerID="5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"
Nov 03 16:03:23.259253 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.259211    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"} err="failed to get container status \"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b\": rpc error: code = NotFound desc = could not find container \"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b\": container with ID starting with 5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b not found: ID does not exist"
Nov 03 16:03:23.259253 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.259246    6506 scope.go:122] "RemoveContainer" containerID="57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"
Nov 03 16:03:23.259476 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:23.259462    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f\": container with ID starting with 57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f not found: ID does not exist" containerID="57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"
Nov 03 16:03:23.259516 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.259482    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"} err="failed to get container status \"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f\": rpc error: code = NotFound desc = could not find container \"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f\": container with ID starting with 57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f not found: ID does not exist"
Nov 03 16:03:23.259516 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.259496    6506 scope.go:122] "RemoveContainer" containerID="fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"
Nov 03 16:03:23.259701 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:23.259685    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7\": container with ID starting with fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7 not found: ID does not exist" containerID="fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"
Nov 03 16:03:23.259736 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.259705    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"} err="failed to get container status \"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7\": rpc error: code = NotFound desc = could not find container \"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7\": container with ID starting with fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7 not found: ID does not exist"
Nov 03 16:03:23.259736 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.259718    6506 scope.go:122] "RemoveContainer" containerID="70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"
Nov 03 16:03:23.260170 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:23.260154    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a\": container with ID starting with 70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a not found: ID does not exist" containerID="70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"
Nov 03 16:03:23.260205 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.260173    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"} err="failed to get container status \"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a\": rpc error: code = NotFound desc = could not find container \"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a\": container with ID starting with 70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a not found: ID does not exist"
Nov 03 16:03:23.260205 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.260185    6506 scope.go:122] "RemoveContainer" containerID="bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"
Nov 03 16:03:23.260420 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:23.260401    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89\": container with ID starting with bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89 not found: ID does not exist" containerID="bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"
Nov 03 16:03:23.260465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.260425    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"} err="failed to get container status \"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89\": rpc error: code = NotFound desc = could not find container \"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89\": container with ID starting with bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89 not found: ID does not exist"
Nov 03 16:03:23.260465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.260443    6506 scope.go:122] "RemoveContainer" containerID="8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"
Nov 03 16:03:23.260658 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:23.260642    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069\": container with ID starting with 8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069 not found: ID does not exist" containerID="8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"
Nov 03 16:03:23.260692 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.260661    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"} err="failed to get container status \"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069\": rpc error: code = NotFound desc = could not find container \"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069\": container with ID starting with 8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069 not found: ID does not exist"
Nov 03 16:03:23.260692 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.260672    6506 scope.go:122] "RemoveContainer" containerID="916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9"
Nov 03 16:03:23.260888 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:23.260872    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9\": container with ID starting with 916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9 not found: ID does not exist" containerID="916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9"
Nov 03 16:03:23.260930 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.260892    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9"} err="failed to get container status \"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9\": rpc error: code = NotFound desc = could not find container \"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9\": container with ID starting with 916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9 not found: ID does not exist"
Nov 03 16:03:23.260930 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.260905    6506 scope.go:122] "RemoveContainer" containerID="5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"
Nov 03 16:03:23.261153 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.261137    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"} err="failed to get container status \"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b\": rpc error: code = NotFound desc = could not find container \"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b\": container with ID starting with 5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b not found: ID does not exist"
Nov 03 16:03:23.261192 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.261154    6506 scope.go:122] "RemoveContainer" containerID="57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"
Nov 03 16:03:23.261369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.261352    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"} err="failed to get container status \"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f\": rpc error: code = NotFound desc = could not find container \"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f\": container with ID starting with 57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f not found: ID does not exist"
Nov 03 16:03:23.261369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.261369    6506 scope.go:122] "RemoveContainer" containerID="fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"
Nov 03 16:03:23.261572 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.261555    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"} err="failed to get container status \"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7\": rpc error: code = NotFound desc = could not find container \"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7\": container with ID starting with fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7 not found: ID does not exist"
Nov 03 16:03:23.261608 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.261572    6506 scope.go:122] "RemoveContainer" containerID="70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"
Nov 03 16:03:23.261786 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.261767    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"} err="failed to get container status \"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a\": rpc error: code = NotFound desc = could not find container \"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a\": container with ID starting with 70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a not found: ID does not exist"
Nov 03 16:03:23.261853 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.261810    6506 scope.go:122] "RemoveContainer" containerID="bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"
Nov 03 16:03:23.262044 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.262028    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"} err="failed to get container status \"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89\": rpc error: code = NotFound desc = could not find container \"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89\": container with ID starting with bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89 not found: ID does not exist"
Nov 03 16:03:23.262044 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.262043    6506 scope.go:122] "RemoveContainer" containerID="8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"
Nov 03 16:03:23.262234 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.262220    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"} err="failed to get container status \"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069\": rpc error: code = NotFound desc = could not find container \"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069\": container with ID starting with 8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069 not found: ID does not exist"
Nov 03 16:03:23.262234 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.262232    6506 scope.go:122] "RemoveContainer" containerID="916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9"
Nov 03 16:03:23.262435 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.262418    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9"} err="failed to get container status \"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9\": rpc error: code = NotFound desc = could not find container \"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9\": container with ID starting with 916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9 not found: ID does not exist"
Nov 03 16:03:23.262483 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.262435    6506 scope.go:122] "RemoveContainer" containerID="5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"
Nov 03 16:03:23.262634 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.262620    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"} err="failed to get container status \"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b\": rpc error: code = NotFound desc = could not find container \"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b\": container with ID starting with 5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b not found: ID does not exist"
Nov 03 16:03:23.262681 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.262634    6506 scope.go:122] "RemoveContainer" containerID="57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"
Nov 03 16:03:23.262864 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.262848    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"} err="failed to get container status \"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f\": rpc error: code = NotFound desc = could not find container \"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f\": container with ID starting with 57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f not found: ID does not exist"
Nov 03 16:03:23.262920 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.262864    6506 scope.go:122] "RemoveContainer" containerID="fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"
Nov 03 16:03:23.263078 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.263063    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"} err="failed to get container status \"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7\": rpc error: code = NotFound desc = could not find container \"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7\": container with ID starting with fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7 not found: ID does not exist"
Nov 03 16:03:23.263127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.263078    6506 scope.go:122] "RemoveContainer" containerID="70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"
Nov 03 16:03:23.263296 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.263278    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"} err="failed to get container status \"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a\": rpc error: code = NotFound desc = could not find container \"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a\": container with ID starting with 70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a not found: ID does not exist"
Nov 03 16:03:23.263337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.263297    6506 scope.go:122] "RemoveContainer" containerID="bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"
Nov 03 16:03:23.263518 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.263500    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"} err="failed to get container status \"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89\": rpc error: code = NotFound desc = could not find container \"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89\": container with ID starting with bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89 not found: ID does not exist"
Nov 03 16:03:23.263555 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.263519    6506 scope.go:122] "RemoveContainer" containerID="8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"
Nov 03 16:03:23.263719 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.263696    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"} err="failed to get container status \"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069\": rpc error: code = NotFound desc = could not find container \"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069\": container with ID starting with 8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069 not found: ID does not exist"
Nov 03 16:03:23.263719 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.263717    6506 scope.go:122] "RemoveContainer" containerID="916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9"
Nov 03 16:03:23.263933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.263917    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9"} err="failed to get container status \"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9\": rpc error: code = NotFound desc = could not find container \"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9\": container with ID starting with 916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9 not found: ID does not exist"
Nov 03 16:03:23.263975 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.263934    6506 scope.go:122] "RemoveContainer" containerID="5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"
Nov 03 16:03:23.264131 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.264114    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"} err="failed to get container status \"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b\": rpc error: code = NotFound desc = could not find container \"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b\": container with ID starting with 5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b not found: ID does not exist"
Nov 03 16:03:23.264173 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.264131    6506 scope.go:122] "RemoveContainer" containerID="57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"
Nov 03 16:03:23.264311 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.264289    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"} err="failed to get container status \"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f\": rpc error: code = NotFound desc = could not find container \"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f\": container with ID starting with 57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f not found: ID does not exist"
Nov 03 16:03:23.264311 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.264308    6506 scope.go:122] "RemoveContainer" containerID="fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"
Nov 03 16:03:23.264531 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.264513    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"} err="failed to get container status \"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7\": rpc error: code = NotFound desc = could not find container \"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7\": container with ID starting with fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7 not found: ID does not exist"
Nov 03 16:03:23.264577 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.264531    6506 scope.go:122] "RemoveContainer" containerID="70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"
Nov 03 16:03:23.264724 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.264710    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"} err="failed to get container status \"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a\": rpc error: code = NotFound desc = could not find container \"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a\": container with ID starting with 70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a not found: ID does not exist"
Nov 03 16:03:23.264771 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.264724    6506 scope.go:122] "RemoveContainer" containerID="bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"
Nov 03 16:03:23.264964 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.264948    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"} err="failed to get container status \"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89\": rpc error: code = NotFound desc = could not find container \"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89\": container with ID starting with bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89 not found: ID does not exist"
Nov 03 16:03:23.264964 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.264964    6506 scope.go:122] "RemoveContainer" containerID="8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"
Nov 03 16:03:23.265176 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.265160    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"} err="failed to get container status \"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069\": rpc error: code = NotFound desc = could not find container \"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069\": container with ID starting with 8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069 not found: ID does not exist"
Nov 03 16:03:23.265222 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.265176    6506 scope.go:122] "RemoveContainer" containerID="916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9"
Nov 03 16:03:23.265408 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.265389    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9"} err="failed to get container status \"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9\": rpc error: code = NotFound desc = could not find container \"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9\": container with ID starting with 916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9 not found: ID does not exist"
Nov 03 16:03:23.265455 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.265408    6506 scope.go:122] "RemoveContainer" containerID="5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"
Nov 03 16:03:23.265638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.265619    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"} err="failed to get container status \"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b\": rpc error: code = NotFound desc = could not find container \"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b\": container with ID starting with 5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b not found: ID does not exist"
Nov 03 16:03:23.265691 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.265638    6506 scope.go:122] "RemoveContainer" containerID="57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"
Nov 03 16:03:23.265872 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.265856    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"} err="failed to get container status \"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f\": rpc error: code = NotFound desc = could not find container \"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f\": container with ID starting with 57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f not found: ID does not exist"
Nov 03 16:03:23.265872 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.265871    6506 scope.go:122] "RemoveContainer" containerID="fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"
Nov 03 16:03:23.266098 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.266081    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"} err="failed to get container status \"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7\": rpc error: code = NotFound desc = could not find container \"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7\": container with ID starting with fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7 not found: ID does not exist"
Nov 03 16:03:23.266144 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.266099    6506 scope.go:122] "RemoveContainer" containerID="70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"
Nov 03 16:03:23.266301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.266284    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"} err="failed to get container status \"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a\": rpc error: code = NotFound desc = could not find container \"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a\": container with ID starting with 70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a not found: ID does not exist"
Nov 03 16:03:23.266337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.266302    6506 scope.go:122] "RemoveContainer" containerID="bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"
Nov 03 16:03:23.266522 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.266500    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"} err="failed to get container status \"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89\": rpc error: code = NotFound desc = could not find container \"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89\": container with ID starting with bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89 not found: ID does not exist"
Nov 03 16:03:23.266522 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.266520    6506 scope.go:122] "RemoveContainer" containerID="8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"
Nov 03 16:03:23.266754 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.266737    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"} err="failed to get container status \"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069\": rpc error: code = NotFound desc = could not find container \"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069\": container with ID starting with 8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069 not found: ID does not exist"
Nov 03 16:03:23.266862 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.266754    6506 scope.go:122] "RemoveContainer" containerID="916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9"
Nov 03 16:03:23.266969 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.266950    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9"} err="failed to get container status \"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9\": rpc error: code = NotFound desc = could not find container \"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9\": container with ID starting with 916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9 not found: ID does not exist"
Nov 03 16:03:23.267052 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.266971    6506 scope.go:122] "RemoveContainer" containerID="5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"
Nov 03 16:03:23.267201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.267186    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b"} err="failed to get container status \"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b\": rpc error: code = NotFound desc = could not find container \"5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b\": container with ID starting with 5b5da825a666775d5504188ae98f6a42843197b8e54bdd61e7c8c5e8811ac99b not found: ID does not exist"
Nov 03 16:03:23.267242 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.267201    6506 scope.go:122] "RemoveContainer" containerID="57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"
Nov 03 16:03:23.267425 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.267408    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f"} err="failed to get container status \"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f\": rpc error: code = NotFound desc = could not find container \"57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f\": container with ID starting with 57a952abd451c1b8364436f768d88ac4601700acd15ca7162691072b28b7a07f not found: ID does not exist"
Nov 03 16:03:23.267467 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.267425    6506 scope.go:122] "RemoveContainer" containerID="fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"
Nov 03 16:03:23.267618 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.267603    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7"} err="failed to get container status \"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7\": rpc error: code = NotFound desc = could not find container \"fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7\": container with ID starting with fe46dc1352ce3c2fb3967bcf547e096e1228eaa281a30da4005e6f6a07c52be7 not found: ID does not exist"
Nov 03 16:03:23.267659 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.267617    6506 scope.go:122] "RemoveContainer" containerID="70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"
Nov 03 16:03:23.267827 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.267810    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a"} err="failed to get container status \"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a\": rpc error: code = NotFound desc = could not find container \"70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a\": container with ID starting with 70528e73ef8e64af87fadb28c8c8254a7973707ab72a42d1a4fbcf6d01fd378a not found: ID does not exist"
Nov 03 16:03:23.267873 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.267827    6506 scope.go:122] "RemoveContainer" containerID="bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"
Nov 03 16:03:23.268053 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.268037    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89"} err="failed to get container status \"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89\": rpc error: code = NotFound desc = could not find container \"bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89\": container with ID starting with bae1f88c78d493c4f7946dfb3ce44b294b478a7bc471dfd66890232d0fe10b89 not found: ID does not exist"
Nov 03 16:03:23.268094 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.268054    6506 scope.go:122] "RemoveContainer" containerID="8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"
Nov 03 16:03:23.268264 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.268246    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069"} err="failed to get container status \"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069\": rpc error: code = NotFound desc = could not find container \"8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069\": container with ID starting with 8f322bb63dd7d90646a0b0d63cf1d0c9244e56a7e906778d4bec49d64e3a8069 not found: ID does not exist"
Nov 03 16:03:23.268301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.268265    6506 scope.go:122] "RemoveContainer" containerID="916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9"
Nov 03 16:03:23.268479 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.268462    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9"} err="failed to get container status \"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9\": rpc error: code = NotFound desc = could not find container \"916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9\": container with ID starting with 916fa0d4b3a98e179ddeea97546774b35d70b7a522884ff3e0d8b5d24eb2f6f9 not found: ID does not exist"
Nov 03 16:03:23.308308 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308284    6506 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-db\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308308 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308304    6506 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-web-config\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308314    6506 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-kube-rbac-proxy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308324    6506 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308333    6506 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308343    6506 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308352    6506 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-tls-assets\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308360    6506 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308368    6506 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308376    6506 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-metrics-client-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308385    6506 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308393    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfqjb\" (UniqueName: \"kubernetes.io/projected/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-kube-api-access-kfqjb\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308402    6506 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308410    6506 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308419    6506 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308428    6506 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-metrics-client-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308825 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308436    6506 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-config-out\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308825 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308445    6506 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-secret-grpc-tls\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.308825 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.308453    6506 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/587c9773-ff2c-4c1d-a4ed-f51fc12ee704-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:23.537935 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.537908    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 16:03:23.541779 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.541737    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 16:03:23.584684 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.584657    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 16:03:23.584994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.584978    6506 cpu_manager.go:414] "containerMap: removing container" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="prometheus"
Nov 03 16:03:23.585075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.584996    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="prometheus"
Nov 03 16:03:23.585075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585008    6506 cpu_manager.go:414] "containerMap: removing container" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="kube-rbac-proxy"
Nov 03 16:03:23.585075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585014    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="kube-rbac-proxy"
Nov 03 16:03:23.585075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585026    6506 cpu_manager.go:414] "containerMap: removing container" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="kube-rbac-proxy-web"
Nov 03 16:03:23.585075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585032    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="kube-rbac-proxy-web"
Nov 03 16:03:23.585075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585038    6506 cpu_manager.go:414] "containerMap: removing container" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="thanos-sidecar"
Nov 03 16:03:23.585075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585044    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="thanos-sidecar"
Nov 03 16:03:23.585075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585051    6506 cpu_manager.go:414] "containerMap: removing container" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="kube-rbac-proxy-thanos"
Nov 03 16:03:23.585075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585056    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="kube-rbac-proxy-thanos"
Nov 03 16:03:23.585075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585062    6506 cpu_manager.go:414] "containerMap: removing container" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="config-reloader"
Nov 03 16:03:23.585075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585067    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="config-reloader"
Nov 03 16:03:23.585075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585074    6506 cpu_manager.go:414] "containerMap: removing container" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="init-config-reloader"
Nov 03 16:03:23.585075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585079    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="init-config-reloader"
Nov 03 16:03:23.585545 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585128    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="prometheus"
Nov 03 16:03:23.585545 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585136    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="kube-rbac-proxy"
Nov 03 16:03:23.585545 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585142    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="kube-rbac-proxy-web"
Nov 03 16:03:23.585545 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585147    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="thanos-sidecar"
Nov 03 16:03:23.585545 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585152    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="kube-rbac-proxy-thanos"
Nov 03 16:03:23.585545 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585157    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="config-reloader"
Nov 03 16:03:23.585545 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.585161    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" containerName="init-config-reloader"
Nov 03 16:03:23.588689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.588674    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.598653 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.598632    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 16:03:23.598749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.598719    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 16:03:23.598922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.598906    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 16:03:23.601020 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.600998    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 16:03:23.601128 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.601049    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 16:03:23.601128 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.601052    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 16:03:23.601128 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.601069    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-duq0m103irmgu\""
Nov 03 16:03:23.601128 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.601076    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8qdq8\""
Nov 03 16:03:23.601672 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.601656    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 16:03:23.601748 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.601660    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 16:03:23.602060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.602046    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 16:03:23.602109 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.602081    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 16:03:23.602160 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.602107    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 16:03:23.602160 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.602131    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 16:03:23.602279 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.602263    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 16:03:23.607535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.607303    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 16:03:23.710886 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.710864    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711003 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.710892    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711003 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.710911    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711003 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.710928    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711003 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.710984    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.711037    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.711068    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.711138    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.711156    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.711203    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.711295    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.711316    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-config\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.711334    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-965qx\" (UniqueName: \"kubernetes.io/projected/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-kube-api-access-965qx\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711542 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.711361    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711542 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.711398    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711542 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.711457    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711542 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.711486    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711542 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.711508    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711712 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.711559    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.711712 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.711583    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.728149 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.728129    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 16:03:23.812386 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812320    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812386 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812357    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812419    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812457    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812476    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812495    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812510    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812524    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812555    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812591    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812614    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812637    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812673    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812700    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-config\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812728    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-965qx\" (UniqueName: \"kubernetes.io/projected/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-kube-api-access-965qx\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812757    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812780    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812832    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812669    6506 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 16:03:23.812932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812859    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.812932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.812898    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.813466 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.813435    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.813564 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.813537    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.813736 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.813706    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.813922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.813900    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.813999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.813949    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.814419 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.814387    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.815530 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.815498    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.815988 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.815960    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.816886 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.816441    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.816886 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.816687    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.816886 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.816825    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-config\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.817112 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.816918    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.817112 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.816926    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.817112 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.816988    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.817922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.817894    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.818419 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.818394    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.818595 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.818577    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.818664 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.818640    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.819177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.819161    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.828005 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.827983    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-965qx\" (UniqueName: \"kubernetes.io/projected/838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc-kube-api-access-965qx\") pod \"prometheus-k8s-0\" (UID: \"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:23.891465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.891443    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:23.904339 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:23.904320    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:24.051207 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:24.051174    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:24.100447 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:24.100382    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="587c9773-ff2c-4c1d-a4ed-f51fc12ee704" path="/var/lib/kubelet/pods/587c9773-ff2c-4c1d-a4ed-f51fc12ee704/volumes"
Nov 03 16:03:24.202749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:24.202723    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 16:03:24.206468 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:03:24.206434    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838d9c9d_ad7c_45f1_a71e_dbf1b9f2e1cc.slice/crio-6f222942a30b1b3fb12ecdafdc100a46cf56c3f467ff15652721abf0f8658c6c WatchSource:0}: Error finding container 6f222942a30b1b3fb12ecdafdc100a46cf56c3f467ff15652721abf0f8658c6c: Status 404 returned error can't find the container with id 6f222942a30b1b3fb12ecdafdc100a46cf56c3f467ff15652721abf0f8658c6c
Nov 03 16:03:24.207421 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:24.207387    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:24.217647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:24.217622    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc","Type":"ContainerStarted","Data":"6f222942a30b1b3fb12ecdafdc100a46cf56c3f467ff15652721abf0f8658c6c"}
Nov 03 16:03:25.222049 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:25.222014    6506 generic.go:356] "Generic (PLEG): container finished" podID="838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc" containerID="75a294893c1231589d8ae14fa5c8c368506efe67f8dc17ecb6b8362e20833101" exitCode=0
Nov 03 16:03:25.222400 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:25.222107    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc","Type":"ContainerDied","Data":"75a294893c1231589d8ae14fa5c8c368506efe67f8dc17ecb6b8362e20833101"}
Nov 03 16:03:26.227130 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:26.227091    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc","Type":"ContainerStarted","Data":"26ec6a5951bdf1283af1fc32df19c2cb9e17d4bfaffe36fabdb3bf9b81f5fab9"}
Nov 03 16:03:26.227501 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:26.227173    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc","Type":"ContainerStarted","Data":"cc8f5a4eb8dc62603b10eebab53d50ff1a3fa7860ae2519e23561125d28a5a25"}
Nov 03 16:03:26.227501 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:26.227216    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc","Type":"ContainerStarted","Data":"53205e326d6608aeeb34664ec4006a77471eee4f0e625b5b83c885c271dff0ef"}
Nov 03 16:03:26.227501 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:26.227258    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc","Type":"ContainerStarted","Data":"4c36180b39574f98742eaa1a851cf13fda4165b22a484fa8dc37872c65cc1f9f"}
Nov 03 16:03:26.227501 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:26.227294    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc","Type":"ContainerStarted","Data":"b85f1c1abd5ebc2367372b0744b183f2bc6a5a1974beb26b221e8bf8a73b64eb"}
Nov 03 16:03:26.227501 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:26.227329    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc","Type":"ContainerStarted","Data":"2670056b346eb4dab744cfcddff70fbbf0f1ce976a7accbfefe2dbab54faeb78"}
Nov 03 16:03:26.270332 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:26.270273    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.254221403 podStartE2EDuration="3.270259129s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.037726ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:03:26.268432081 +0000 UTC m=+20.281830262" watchObservedRunningTime="2026-11-03 16:03:26.270259129 +0000 UTC m=+20.283657276"
Nov 03 16:03:28.794884 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:28.794847    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:03:28.905422 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:28.905390    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:38.899950 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:38.899916    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:39.051689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:39.051657    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:03:41.361645 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:41.361608    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2ht7q"
Nov 03 16:03:55.778686 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:55.778654    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:55.795277 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:55.795253    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:56.333993 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:03:56.333970    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 16:03:58.794855 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:03:58.794819    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:04:03.670027 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:04:03.669990    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:04:03.822250 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:04:03.822216    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:04:08.360901 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:04:08.360865    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:04:10.454947 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:04:10.454915    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:04:24.198447 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:04:24.198400    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:04:24.362832 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:04:24.362779    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:04:28.795789 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:04:28.795755    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:04:38.961066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:04:38.961037    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:04:58.795344 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:04:58.795263    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:05:08.425521 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:05:08.425487    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:05:21.528934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:05:21.528900    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:05:28.795645 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:05:28.795604    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:05:58.795609 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:05:58.795573    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:06:06.680010 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:06:06.679976    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:06:06.833199 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:06:06.833163    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:06:08.515055 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:06:08.515024    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:06:28.794979 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:06:28.794944    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:06:31.594201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:06:31.594168    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:06:31.743765 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:06:31.743729    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:06:58.794840 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:06:58.794779    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:07:08.556452 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:07:08.556415    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:07:28.794684 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:07:28.794644    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:07:58.794913 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:07:58.794879    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:08:08.550213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:08:08.550170    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:08:28.795274 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:08:28.795230    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:08:58.794735 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:08:58.794700    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:09:08.671108 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:09:08.671065    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:09:28.794492 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:09:28.794455    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:09:58.794984 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:09:58.794910    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:10:08.709002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:08.708959    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:10:28.634273 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:28.634234    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:10:28.795591 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:10:28.795553    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:10:28.822708 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:28.822687    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:10:30.146486 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:30.146446    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:10:30.319994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:30.319961    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:10:30.986988 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:30.986954    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:10:31.579471 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:31.579436    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:10:33.854276 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:33.854238    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"]
Nov 03 16:10:33.857908 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:33.857884    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"
Nov 03 16:10:33.861968 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:33.861947    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 16:10:33.861968 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:33.861957    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 16:10:33.863125 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:33.863096    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-gv4cs\""
Nov 03 16:10:33.877330 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:33.877310    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"]
Nov 03 16:10:33.936499 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:33.936471    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0704aaa3-8df1-41c9-8df3-0975a90ec44a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn\" (UID: \"0704aaa3-8df1-41c9-8df3-0975a90ec44a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"
Nov 03 16:10:33.936651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:33.936524    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztpxt\" (UniqueName: \"kubernetes.io/projected/0704aaa3-8df1-41c9-8df3-0975a90ec44a-kube-api-access-ztpxt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn\" (UID: \"0704aaa3-8df1-41c9-8df3-0975a90ec44a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"
Nov 03 16:10:33.936651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:33.936629    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0704aaa3-8df1-41c9-8df3-0975a90ec44a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn\" (UID: \"0704aaa3-8df1-41c9-8df3-0975a90ec44a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"
Nov 03 16:10:34.037526 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.037493    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztpxt\" (UniqueName: \"kubernetes.io/projected/0704aaa3-8df1-41c9-8df3-0975a90ec44a-kube-api-access-ztpxt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn\" (UID: \"0704aaa3-8df1-41c9-8df3-0975a90ec44a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"
Nov 03 16:10:34.037703 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.037547    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0704aaa3-8df1-41c9-8df3-0975a90ec44a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn\" (UID: \"0704aaa3-8df1-41c9-8df3-0975a90ec44a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"
Nov 03 16:10:34.037769 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.037712    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0704aaa3-8df1-41c9-8df3-0975a90ec44a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn\" (UID: \"0704aaa3-8df1-41c9-8df3-0975a90ec44a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"
Nov 03 16:10:34.037888 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.037871    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0704aaa3-8df1-41c9-8df3-0975a90ec44a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn\" (UID: \"0704aaa3-8df1-41c9-8df3-0975a90ec44a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"
Nov 03 16:10:34.038023 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.038004    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0704aaa3-8df1-41c9-8df3-0975a90ec44a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn\" (UID: \"0704aaa3-8df1-41c9-8df3-0975a90ec44a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"
Nov 03 16:10:34.060081 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.060052    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztpxt\" (UniqueName: \"kubernetes.io/projected/0704aaa3-8df1-41c9-8df3-0975a90ec44a-kube-api-access-ztpxt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn\" (UID: \"0704aaa3-8df1-41c9-8df3-0975a90ec44a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"
Nov 03 16:10:34.173164 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.173094    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"
Nov 03 16:10:34.444624 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.444548    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"]
Nov 03 16:10:34.448289 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:10:34.448265    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0704aaa3_8df1_41c9_8df3_0975a90ec44a.slice/crio-06b65e44c72eda6e838130f641e6167a7946cfd621a3f1c82fee7d912ec9fabf WatchSource:0}: Error finding container 06b65e44c72eda6e838130f641e6167a7946cfd621a3f1c82fee7d912ec9fabf: Status 404 returned error can't find the container with id 06b65e44c72eda6e838130f641e6167a7946cfd621a3f1c82fee7d912ec9fabf
Nov 03 16:10:34.450354 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.450339    6506 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:10:34.583719 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.583688    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"]
Nov 03 16:10:34.588460 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.588437    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"
Nov 03 16:10:34.592502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.592396    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn" event={"ID":"0704aaa3-8df1-41c9-8df3-0975a90ec44a","Type":"ContainerStarted","Data":"06b65e44c72eda6e838130f641e6167a7946cfd621a3f1c82fee7d912ec9fabf"}
Nov 03 16:10:34.593551 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.593532    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-d92q7\""
Nov 03 16:10:34.597761 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.597741    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"]
Nov 03 16:10:34.702388 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.702323    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"]
Nov 03 16:10:34.707315 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.707290    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"
Nov 03 16:10:34.715593 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.715553    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-6rhjp\""
Nov 03 16:10:34.717243 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.717222    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"]
Nov 03 16:10:34.742321 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.742296    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84f6338c-87ba-4910-b41a-e8d4f29b9d88-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v\" (UID: \"84f6338c-87ba-4910-b41a-e8d4f29b9d88\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"
Nov 03 16:10:34.742416 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.742340    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84f6338c-87ba-4910-b41a-e8d4f29b9d88-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v\" (UID: \"84f6338c-87ba-4910-b41a-e8d4f29b9d88\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"
Nov 03 16:10:34.742416 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.742377    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqc7h\" (UniqueName: \"kubernetes.io/projected/84f6338c-87ba-4910-b41a-e8d4f29b9d88-kube-api-access-sqc7h\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v\" (UID: \"84f6338c-87ba-4910-b41a-e8d4f29b9d88\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"
Nov 03 16:10:34.783392 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.783367    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"]
Nov 03 16:10:34.786612 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.786597    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"
Nov 03 16:10:34.790451 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.790429    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-g8pn8\""
Nov 03 16:10:34.799304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.799281    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"]
Nov 03 16:10:34.843563 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.843534    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84f6338c-87ba-4910-b41a-e8d4f29b9d88-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v\" (UID: \"84f6338c-87ba-4910-b41a-e8d4f29b9d88\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"
Nov 03 16:10:34.843677 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.843579    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6ksn\" (UniqueName: \"kubernetes.io/projected/6756018e-938d-4549-b33c-d80c5eebd2a9-kube-api-access-n6ksn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx\" (UID: \"6756018e-938d-4549-b33c-d80c5eebd2a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"
Nov 03 16:10:34.843677 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.843604    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84f6338c-87ba-4910-b41a-e8d4f29b9d88-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v\" (UID: \"84f6338c-87ba-4910-b41a-e8d4f29b9d88\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"
Nov 03 16:10:34.843747 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.843689    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6756018e-938d-4549-b33c-d80c5eebd2a9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx\" (UID: \"6756018e-938d-4549-b33c-d80c5eebd2a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"
Nov 03 16:10:34.843747 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.843719    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqc7h\" (UniqueName: \"kubernetes.io/projected/84f6338c-87ba-4910-b41a-e8d4f29b9d88-kube-api-access-sqc7h\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v\" (UID: \"84f6338c-87ba-4910-b41a-e8d4f29b9d88\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"
Nov 03 16:10:34.843839 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.843751    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6756018e-938d-4549-b33c-d80c5eebd2a9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx\" (UID: \"6756018e-938d-4549-b33c-d80c5eebd2a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"
Nov 03 16:10:34.843955 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.843937    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84f6338c-87ba-4910-b41a-e8d4f29b9d88-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v\" (UID: \"84f6338c-87ba-4910-b41a-e8d4f29b9d88\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"
Nov 03 16:10:34.843994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.843975    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84f6338c-87ba-4910-b41a-e8d4f29b9d88-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v\" (UID: \"84f6338c-87ba-4910-b41a-e8d4f29b9d88\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"
Nov 03 16:10:34.856257 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.856234    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqc7h\" (UniqueName: \"kubernetes.io/projected/84f6338c-87ba-4910-b41a-e8d4f29b9d88-kube-api-access-sqc7h\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v\" (UID: \"84f6338c-87ba-4910-b41a-e8d4f29b9d88\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"
Nov 03 16:10:34.877178 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.877159    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"]
Nov 03 16:10:34.880451 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.880436    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"
Nov 03 16:10:34.890620 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.890602    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-xlqdk\""
Nov 03 16:10:34.890906 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.890889    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"]
Nov 03 16:10:34.905761 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.905739    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"
Nov 03 16:10:34.944350 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.944320    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6756018e-938d-4549-b33c-d80c5eebd2a9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx\" (UID: \"6756018e-938d-4549-b33c-d80c5eebd2a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"
Nov 03 16:10:34.944523 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.944375    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6756018e-938d-4549-b33c-d80c5eebd2a9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx\" (UID: \"6756018e-938d-4549-b33c-d80c5eebd2a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"
Nov 03 16:10:34.944523 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.944425    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5808de3e-b030-4290-98de-a2b476faac9b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt\" (UID: \"5808de3e-b030-4290-98de-a2b476faac9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"
Nov 03 16:10:34.944523 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.944459    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6ksn\" (UniqueName: \"kubernetes.io/projected/6756018e-938d-4549-b33c-d80c5eebd2a9-kube-api-access-n6ksn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx\" (UID: \"6756018e-938d-4549-b33c-d80c5eebd2a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"
Nov 03 16:10:34.944523 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.944513    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5808de3e-b030-4290-98de-a2b476faac9b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt\" (UID: \"5808de3e-b030-4290-98de-a2b476faac9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"
Nov 03 16:10:34.944725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.944540    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcmvz\" (UniqueName: \"kubernetes.io/projected/5808de3e-b030-4290-98de-a2b476faac9b-kube-api-access-hcmvz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt\" (UID: \"5808de3e-b030-4290-98de-a2b476faac9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"
Nov 03 16:10:34.945043 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.945006    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6756018e-938d-4549-b33c-d80c5eebd2a9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx\" (UID: \"6756018e-938d-4549-b33c-d80c5eebd2a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"
Nov 03 16:10:34.945164 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.945087    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6756018e-938d-4549-b33c-d80c5eebd2a9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx\" (UID: \"6756018e-938d-4549-b33c-d80c5eebd2a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"
Nov 03 16:10:34.959537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:34.959517    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6ksn\" (UniqueName: \"kubernetes.io/projected/6756018e-938d-4549-b33c-d80c5eebd2a9-kube-api-access-n6ksn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx\" (UID: \"6756018e-938d-4549-b33c-d80c5eebd2a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"
Nov 03 16:10:35.023690 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.023661    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"
Nov 03 16:10:35.046035 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.045999    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c501915-1a3d-4309-bb99-54f5f3ec0294-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6\" (UID: \"4c501915-1a3d-4309-bb99-54f5f3ec0294\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"
Nov 03 16:10:35.046139 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.046079    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5808de3e-b030-4290-98de-a2b476faac9b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt\" (UID: \"5808de3e-b030-4290-98de-a2b476faac9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"
Nov 03 16:10:35.046185 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.046145    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5808de3e-b030-4290-98de-a2b476faac9b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt\" (UID: \"5808de3e-b030-4290-98de-a2b476faac9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"
Nov 03 16:10:35.046185 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.046172    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcmvz\" (UniqueName: \"kubernetes.io/projected/5808de3e-b030-4290-98de-a2b476faac9b-kube-api-access-hcmvz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt\" (UID: \"5808de3e-b030-4290-98de-a2b476faac9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"
Nov 03 16:10:35.046270 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.046200    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxxd\" (UniqueName: \"kubernetes.io/projected/4c501915-1a3d-4309-bb99-54f5f3ec0294-kube-api-access-pnxxd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6\" (UID: \"4c501915-1a3d-4309-bb99-54f5f3ec0294\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"
Nov 03 16:10:35.046270 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.046262    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c501915-1a3d-4309-bb99-54f5f3ec0294-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6\" (UID: \"4c501915-1a3d-4309-bb99-54f5f3ec0294\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"
Nov 03 16:10:35.046699 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.046676    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5808de3e-b030-4290-98de-a2b476faac9b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt\" (UID: \"5808de3e-b030-4290-98de-a2b476faac9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"
Nov 03 16:10:35.046991 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.046967    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5808de3e-b030-4290-98de-a2b476faac9b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt\" (UID: \"5808de3e-b030-4290-98de-a2b476faac9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"
Nov 03 16:10:35.078002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.077969    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcmvz\" (UniqueName: \"kubernetes.io/projected/5808de3e-b030-4290-98de-a2b476faac9b-kube-api-access-hcmvz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt\" (UID: \"5808de3e-b030-4290-98de-a2b476faac9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"
Nov 03 16:10:35.102910 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.102870    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"
Nov 03 16:10:35.148429 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.147246    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxxd\" (UniqueName: \"kubernetes.io/projected/4c501915-1a3d-4309-bb99-54f5f3ec0294-kube-api-access-pnxxd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6\" (UID: \"4c501915-1a3d-4309-bb99-54f5f3ec0294\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"
Nov 03 16:10:35.148429 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.147323    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c501915-1a3d-4309-bb99-54f5f3ec0294-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6\" (UID: \"4c501915-1a3d-4309-bb99-54f5f3ec0294\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"
Nov 03 16:10:35.148429 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.147358    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c501915-1a3d-4309-bb99-54f5f3ec0294-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6\" (UID: \"4c501915-1a3d-4309-bb99-54f5f3ec0294\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"
Nov 03 16:10:35.148429 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.147746    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c501915-1a3d-4309-bb99-54f5f3ec0294-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6\" (UID: \"4c501915-1a3d-4309-bb99-54f5f3ec0294\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"
Nov 03 16:10:35.148429 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.148386    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c501915-1a3d-4309-bb99-54f5f3ec0294-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6\" (UID: \"4c501915-1a3d-4309-bb99-54f5f3ec0294\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"
Nov 03 16:10:35.161564 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.161493    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxxd\" (UniqueName: \"kubernetes.io/projected/4c501915-1a3d-4309-bb99-54f5f3ec0294-kube-api-access-pnxxd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6\" (UID: \"4c501915-1a3d-4309-bb99-54f5f3ec0294\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"
Nov 03 16:10:35.188174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.188099    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"]
Nov 03 16:10:35.194015 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:10:35.193988    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f6338c_87ba_4910_b41a_e8d4f29b9d88.slice/crio-518e553007a8aa09356d3d641eef0b2743f72b5e258952f639d5af77eea243bb WatchSource:0}: Error finding container 518e553007a8aa09356d3d641eef0b2743f72b5e258952f639d5af77eea243bb: Status 404 returned error can't find the container with id 518e553007a8aa09356d3d641eef0b2743f72b5e258952f639d5af77eea243bb
Nov 03 16:10:35.196508 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.196478    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"
Nov 03 16:10:35.307542 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.307464    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"]
Nov 03 16:10:35.310540 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:10:35.310510    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6756018e_938d_4549_b33c_d80c5eebd2a9.slice/crio-c81bf48bc1f8986c8d28e76faa5cce9f8f70db9a29d8fdacf2a3e14352e97574 WatchSource:0}: Error finding container c81bf48bc1f8986c8d28e76faa5cce9f8f70db9a29d8fdacf2a3e14352e97574: Status 404 returned error can't find the container with id c81bf48bc1f8986c8d28e76faa5cce9f8f70db9a29d8fdacf2a3e14352e97574
Nov 03 16:10:35.315608 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.315582    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:10:35.396899 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.396856    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"]
Nov 03 16:10:35.399822 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:10:35.399767    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5808de3e_b030_4290_98de_a2b476faac9b.slice/crio-45e28d238f22555e878b65d30049a354579c6beff86f45d17a55a178c9bffe3b WatchSource:0}: Error finding container 45e28d238f22555e878b65d30049a354579c6beff86f45d17a55a178c9bffe3b: Status 404 returned error can't find the container with id 45e28d238f22555e878b65d30049a354579c6beff86f45d17a55a178c9bffe3b
Nov 03 16:10:35.476182 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.476115    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"]
Nov 03 16:10:35.477665 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.477639    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:10:35.479682 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:10:35.479657    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c501915_1a3d_4309_bb99_54f5f3ec0294.slice/crio-0201549e2ef7a0f3d6b470b0d1eeff8fc853c2b256ea0cfc1e48a6f5ef8982c4 WatchSource:0}: Error finding container 0201549e2ef7a0f3d6b470b0d1eeff8fc853c2b256ea0cfc1e48a6f5ef8982c4: Status 404 returned error can't find the container with id 0201549e2ef7a0f3d6b470b0d1eeff8fc853c2b256ea0cfc1e48a6f5ef8982c4
Nov 03 16:10:35.529777 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.529745    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"]
Nov 03 16:10:35.535178 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.535153    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.546638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.546614    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-s22x6\""
Nov 03 16:10:35.546769 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.546755    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 16:10:35.546847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.546759    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 16:10:35.546847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.546761    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 16:10:35.550214 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.550052    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"]
Nov 03 16:10:35.598371 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.598338    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx" event={"ID":"6756018e-938d-4549-b33c-d80c5eebd2a9","Type":"ContainerStarted","Data":"c81bf48bc1f8986c8d28e76faa5cce9f8f70db9a29d8fdacf2a3e14352e97574"}
Nov 03 16:10:35.599608 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.599578    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6" event={"ID":"4c501915-1a3d-4309-bb99-54f5f3ec0294","Type":"ContainerStarted","Data":"0201549e2ef7a0f3d6b470b0d1eeff8fc853c2b256ea0cfc1e48a6f5ef8982c4"}
Nov 03 16:10:35.600642 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.600615    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v" event={"ID":"84f6338c-87ba-4910-b41a-e8d4f29b9d88","Type":"ContainerStarted","Data":"518e553007a8aa09356d3d641eef0b2743f72b5e258952f639d5af77eea243bb"}
Nov 03 16:10:35.601898 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.601874    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt" event={"ID":"5808de3e-b030-4290-98de-a2b476faac9b","Type":"ContainerStarted","Data":"45e28d238f22555e878b65d30049a354579c6beff86f45d17a55a178c9bffe3b"}
Nov 03 16:10:35.650941 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.650911    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.651054 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.650966    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.651054 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.650993    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.651054 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.651028    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.651238 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.651067    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.651238 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.651092    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.651238 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.651131    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.651238 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.651158    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96t87\" (UniqueName: \"kubernetes.io/projected/52baf237-4a39-4822-b1b4-4d9a6bd36b85-kube-api-access-96t87\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.651238 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.651200    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.651660 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.651598    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:10:35.752866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.751773    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.752866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.751859    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.752866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.751892    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.752866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.751908    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.752866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.751931    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.752866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.751959    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.752866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.751982    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.752866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.752020    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.752866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.752040    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96t87\" (UniqueName: \"kubernetes.io/projected/52baf237-4a39-4822-b1b4-4d9a6bd36b85-kube-api-access-96t87\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.753472 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.752905    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.753472 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.752973    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.753472 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.752995    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.753782 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.753648    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.753782 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.753741    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.755455 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.755367    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.756978 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.756944    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.765822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.765783    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.767146 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.767121    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96t87\" (UniqueName: \"kubernetes.io/projected/52baf237-4a39-4822-b1b4-4d9a6bd36b85-kube-api-access-96t87\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sdjqz\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:35.823314 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.823272    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:10:35.849327 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:35.849295    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:36.143785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:36.143609    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"]
Nov 03 16:10:36.153806 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:10:36.153752    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52baf237_4a39_4822_b1b4_4d9a6bd36b85.slice/crio-d21e7437e2c673fba2261de6b3abbea00a2ffbe08b64bbdb471d4f3b85db3f88 WatchSource:0}: Error finding container d21e7437e2c673fba2261de6b3abbea00a2ffbe08b64bbdb471d4f3b85db3f88: Status 404 returned error can't find the container with id d21e7437e2c673fba2261de6b3abbea00a2ffbe08b64bbdb471d4f3b85db3f88
Nov 03 16:10:36.611751 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:36.611714    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz" event={"ID":"52baf237-4a39-4822-b1b4-4d9a6bd36b85","Type":"ContainerStarted","Data":"d21e7437e2c673fba2261de6b3abbea00a2ffbe08b64bbdb471d4f3b85db3f88"}
Nov 03 16:10:36.677511 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:36.677416    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"]
Nov 03 16:10:36.683227 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:36.683203    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"
Nov 03 16:10:36.688026 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:36.688000    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-z56d6\""
Nov 03 16:10:36.707015 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:36.706947    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"]
Nov 03 16:10:36.865518 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:36.865409    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvfh7\" (UniqueName: \"kubernetes.io/projected/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-kube-api-access-nvfh7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht\" (UID: \"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"
Nov 03 16:10:36.865682 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:36.865634    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht\" (UID: \"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"
Nov 03 16:10:36.865725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:36.865678    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht\" (UID: \"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"
Nov 03 16:10:36.966424 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:36.966383    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht\" (UID: \"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"
Nov 03 16:10:36.966578 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:36.966497    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvfh7\" (UniqueName: \"kubernetes.io/projected/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-kube-api-access-nvfh7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht\" (UID: \"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"
Nov 03 16:10:36.966578 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:36.966556    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht\" (UID: \"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"
Nov 03 16:10:36.966918 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:36.966894    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht\" (UID: \"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"
Nov 03 16:10:36.967085 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:36.967062    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht\" (UID: \"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"
Nov 03 16:10:36.999824 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:36.999776    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvfh7\" (UniqueName: \"kubernetes.io/projected/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-kube-api-access-nvfh7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht\" (UID: \"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"
Nov 03 16:10:37.004428 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.004404    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"
Nov 03 16:10:37.299807 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.299729    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"]
Nov 03 16:10:37.311043 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:10:37.310956    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c8d92c_0b9e_42c7_9bcb_e626b371f4a4.slice/crio-7b15d57da11faf6eb6cdcf1e2240463685922f17c14a45bec992a4104ad5bb3b WatchSource:0}: Error finding container 7b15d57da11faf6eb6cdcf1e2240463685922f17c14a45bec992a4104ad5bb3b: Status 404 returned error can't find the container with id 7b15d57da11faf6eb6cdcf1e2240463685922f17c14a45bec992a4104ad5bb3b
Nov 03 16:10:37.623003 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.622928    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht" event={"ID":"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4","Type":"ContainerStarted","Data":"7b15d57da11faf6eb6cdcf1e2240463685922f17c14a45bec992a4104ad5bb3b"}
Nov 03 16:10:37.667617 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.667577    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"]
Nov 03 16:10:37.672638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.671879    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"
Nov 03 16:10:37.681730 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.681684    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-f4nqg\""
Nov 03 16:10:37.685633 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.684013    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"]
Nov 03 16:10:37.776444 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.776370    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzzjn\" (UniqueName: \"kubernetes.io/projected/b8136e22-5d4e-460b-b15e-defa2ca183e4-kube-api-access-qzzjn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb\" (UID: \"b8136e22-5d4e-460b-b15e-defa2ca183e4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"
Nov 03 16:10:37.776602 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.776496    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8136e22-5d4e-460b-b15e-defa2ca183e4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb\" (UID: \"b8136e22-5d4e-460b-b15e-defa2ca183e4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"
Nov 03 16:10:37.776602 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.776534    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8136e22-5d4e-460b-b15e-defa2ca183e4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb\" (UID: \"b8136e22-5d4e-460b-b15e-defa2ca183e4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"
Nov 03 16:10:37.878369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.877197    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8136e22-5d4e-460b-b15e-defa2ca183e4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb\" (UID: \"b8136e22-5d4e-460b-b15e-defa2ca183e4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"
Nov 03 16:10:37.878369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.877250    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8136e22-5d4e-460b-b15e-defa2ca183e4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb\" (UID: \"b8136e22-5d4e-460b-b15e-defa2ca183e4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"
Nov 03 16:10:37.878369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.877349    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzzjn\" (UniqueName: \"kubernetes.io/projected/b8136e22-5d4e-460b-b15e-defa2ca183e4-kube-api-access-qzzjn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb\" (UID: \"b8136e22-5d4e-460b-b15e-defa2ca183e4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"
Nov 03 16:10:37.878369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.878057    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8136e22-5d4e-460b-b15e-defa2ca183e4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb\" (UID: \"b8136e22-5d4e-460b-b15e-defa2ca183e4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"
Nov 03 16:10:37.878369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.878285    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8136e22-5d4e-460b-b15e-defa2ca183e4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb\" (UID: \"b8136e22-5d4e-460b-b15e-defa2ca183e4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"
Nov 03 16:10:37.907462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.907434    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzzjn\" (UniqueName: \"kubernetes.io/projected/b8136e22-5d4e-460b-b15e-defa2ca183e4-kube-api-access-qzzjn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb\" (UID: \"b8136e22-5d4e-460b-b15e-defa2ca183e4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"
Nov 03 16:10:37.999287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:37.999227    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"
Nov 03 16:10:38.336585 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:38.336504    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"]
Nov 03 16:10:38.351653 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:10:38.351611    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8136e22_5d4e_460b_b15e_defa2ca183e4.slice/crio-99469a7cf4f92fc8786233dce8d3afc77c8bee9b9ea41315e687ed3b5b6f7207 WatchSource:0}: Error finding container 99469a7cf4f92fc8786233dce8d3afc77c8bee9b9ea41315e687ed3b5b6f7207: Status 404 returned error can't find the container with id 99469a7cf4f92fc8786233dce8d3afc77c8bee9b9ea41315e687ed3b5b6f7207
Nov 03 16:10:38.478917 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:38.478806    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:10:38.632486 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:38.632405    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb" event={"ID":"b8136e22-5d4e-460b-b15e-defa2ca183e4","Type":"ContainerStarted","Data":"99469a7cf4f92fc8786233dce8d3afc77c8bee9b9ea41315e687ed3b5b6f7207"}
Nov 03 16:10:44.061413 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:44.061366    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:10:44.061891 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:44.061454    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:10:44.061891 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:44.061498    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:10:44.691121 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:44.691080    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz" event={"ID":"52baf237-4a39-4822-b1b4-4d9a6bd36b85","Type":"ContainerStarted","Data":"58e19e8ed66f23c5896cde17db008e9264921b1577f320b0d3ca6a554744c5e9"}
Nov 03 16:10:44.718018 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:44.717962    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz" podStartSLOduration=1.8200723189999999 podStartE2EDuration="9.717946766s" totalImagesPullingTime="7.897874447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:44.717118877 +0000 UTC m=+458.730517014" watchObservedRunningTime="2026-11-03 16:10:44.717946766 +0000 UTC m=+458.731344912"
Nov 03 16:10:44.849761 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:44.849725    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:45.856943 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:45.856359    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:46.704915 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:46.704847    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:46.706360 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:46.706137    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:10:48.721056 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:48.718535    6506 generic.go:356] "Generic (PLEG): container finished" podID="0704aaa3-8df1-41c9-8df3-0975a90ec44a" containerID="441e70123760e18fee87439a18e88a99b9e90edf91e316694ff005e55c394e59" exitCode=0
Nov 03 16:10:48.721056 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:48.720568    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn" event={"ID":"0704aaa3-8df1-41c9-8df3-0975a90ec44a","Type":"ContainerDied","Data":"441e70123760e18fee87439a18e88a99b9e90edf91e316694ff005e55c394e59"}
Nov 03 16:10:49.726080 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:49.725976    6506 generic.go:356] "Generic (PLEG): container finished" podID="4c501915-1a3d-4309-bb99-54f5f3ec0294" containerID="007d69e5a738f871a96aa11f78520a56eddafff7467b84856d87c3df0e27ee53" exitCode=0
Nov 03 16:10:49.726453 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:49.726117    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6" event={"ID":"4c501915-1a3d-4309-bb99-54f5f3ec0294","Type":"ContainerDied","Data":"007d69e5a738f871a96aa11f78520a56eddafff7467b84856d87c3df0e27ee53"}
Nov 03 16:10:49.728665 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:49.728609    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v" event={"ID":"84f6338c-87ba-4910-b41a-e8d4f29b9d88","Type":"ContainerStarted","Data":"d11cc149934b6fd49d865147bc654fad0a3ba42b8922b9a2bf8cb1cf4d0d8a2a"}
Nov 03 16:10:49.768262 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:10:49.767722    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f6338c_87ba_4910_b41a_e8d4f29b9d88.slice/crio-conmon-d11cc149934b6fd49d865147bc654fad0a3ba42b8922b9a2bf8cb1cf4d0d8a2a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:10:50.734729 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:50.734698    6506 generic.go:356] "Generic (PLEG): container finished" podID="0704aaa3-8df1-41c9-8df3-0975a90ec44a" containerID="12c82a6b87ff9d21a04008b3a8fe17c9b36c2df1c70570239db9dbbba5899cde" exitCode=0
Nov 03 16:10:50.735112 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:50.734780    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn" event={"ID":"0704aaa3-8df1-41c9-8df3-0975a90ec44a","Type":"ContainerDied","Data":"12c82a6b87ff9d21a04008b3a8fe17c9b36c2df1c70570239db9dbbba5899cde"}
Nov 03 16:10:50.736828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:50.736664    6506 generic.go:356] "Generic (PLEG): container finished" podID="5808de3e-b030-4290-98de-a2b476faac9b" containerID="7bde12803566dd9c93f033fe3ad59b2e9ec1b336b4ac0d1c2dbf8e8a3639c3ab" exitCode=0
Nov 03 16:10:50.736828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:50.736743    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt" event={"ID":"5808de3e-b030-4290-98de-a2b476faac9b","Type":"ContainerDied","Data":"7bde12803566dd9c93f033fe3ad59b2e9ec1b336b4ac0d1c2dbf8e8a3639c3ab"}
Nov 03 16:10:50.738681 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:50.738660    6506 generic.go:356] "Generic (PLEG): container finished" podID="84f6338c-87ba-4910-b41a-e8d4f29b9d88" containerID="d11cc149934b6fd49d865147bc654fad0a3ba42b8922b9a2bf8cb1cf4d0d8a2a" exitCode=0
Nov 03 16:10:50.738761 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:50.738746    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v" event={"ID":"84f6338c-87ba-4910-b41a-e8d4f29b9d88","Type":"ContainerDied","Data":"d11cc149934b6fd49d865147bc654fad0a3ba42b8922b9a2bf8cb1cf4d0d8a2a"}
Nov 03 16:10:50.740474 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:50.740452    6506 generic.go:356] "Generic (PLEG): container finished" podID="44c8d92c-0b9e-42c7-9bcb-e626b371f4a4" containerID="036e2086f4859a0d6b07626a77d3f879ee034b5fda97eb5e79ebf2bcf394755f" exitCode=0
Nov 03 16:10:50.740656 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:50.740482    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht" event={"ID":"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4","Type":"ContainerDied","Data":"036e2086f4859a0d6b07626a77d3f879ee034b5fda97eb5e79ebf2bcf394755f"}
Nov 03 16:10:51.747114 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:51.747026    6506 generic.go:356] "Generic (PLEG): container finished" podID="4c501915-1a3d-4309-bb99-54f5f3ec0294" containerID="90403a325c40466d425f93648fcfe8d876586e8ac8af8c068c9e874c89499843" exitCode=0
Nov 03 16:10:51.747542 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:51.747123    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6" event={"ID":"4c501915-1a3d-4309-bb99-54f5f3ec0294","Type":"ContainerDied","Data":"90403a325c40466d425f93648fcfe8d876586e8ac8af8c068c9e874c89499843"}
Nov 03 16:10:51.749065 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:51.749039    6506 generic.go:356] "Generic (PLEG): container finished" podID="b8136e22-5d4e-460b-b15e-defa2ca183e4" containerID="c7a3f0cf14a33e17db5d6a3eaf7d705938d58384d6691ea644d3ad3e10ab5848" exitCode=0
Nov 03 16:10:51.749183 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:51.749137    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb" event={"ID":"b8136e22-5d4e-460b-b15e-defa2ca183e4","Type":"ContainerDied","Data":"c7a3f0cf14a33e17db5d6a3eaf7d705938d58384d6691ea644d3ad3e10ab5848"}
Nov 03 16:10:51.751208 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:51.751116    6506 generic.go:356] "Generic (PLEG): container finished" podID="6756018e-938d-4549-b33c-d80c5eebd2a9" containerID="28f39576e01be9f8e9092b72919d2147c7d782a12c15bad9b9b5ea82bb673845" exitCode=0
Nov 03 16:10:51.751208 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:51.751169    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx" event={"ID":"6756018e-938d-4549-b33c-d80c5eebd2a9","Type":"ContainerDied","Data":"28f39576e01be9f8e9092b72919d2147c7d782a12c15bad9b9b5ea82bb673845"}
Nov 03 16:10:52.766325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:52.765225    6506 generic.go:356] "Generic (PLEG): container finished" podID="5808de3e-b030-4290-98de-a2b476faac9b" containerID="d5997934036b431cb2db70425a5a6aa8f707a118bd906b11bd703f5488026b05" exitCode=0
Nov 03 16:10:52.766325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:52.765334    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt" event={"ID":"5808de3e-b030-4290-98de-a2b476faac9b","Type":"ContainerDied","Data":"d5997934036b431cb2db70425a5a6aa8f707a118bd906b11bd703f5488026b05"}
Nov 03 16:10:52.770278 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:52.770250    6506 generic.go:356] "Generic (PLEG): container finished" podID="84f6338c-87ba-4910-b41a-e8d4f29b9d88" containerID="58cf8f5155f728e92180d2680bf3a7fecf8dfb71b110c7655a6b3fd38ab80057" exitCode=0
Nov 03 16:10:52.770389 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:52.770314    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v" event={"ID":"84f6338c-87ba-4910-b41a-e8d4f29b9d88","Type":"ContainerDied","Data":"58cf8f5155f728e92180d2680bf3a7fecf8dfb71b110c7655a6b3fd38ab80057"}
Nov 03 16:10:53.779121 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:53.779089    6506 generic.go:356] "Generic (PLEG): container finished" podID="6756018e-938d-4549-b33c-d80c5eebd2a9" containerID="4e9f1a045b78f142fc072e674842f5fcbc2f55ac31748f7cb5d4cf4e0640a09b" exitCode=0
Nov 03 16:10:53.779559 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:53.779188    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx" event={"ID":"6756018e-938d-4549-b33c-d80c5eebd2a9","Type":"ContainerDied","Data":"4e9f1a045b78f142fc072e674842f5fcbc2f55ac31748f7cb5d4cf4e0640a09b"}
Nov 03 16:10:55.801933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:55.801882    6506 generic.go:356] "Generic (PLEG): container finished" podID="44c8d92c-0b9e-42c7-9bcb-e626b371f4a4" containerID="2fc63eb85a4777d3fc73be548732ac7e7fd394e85fa66cfb390a1d54ee5e81b6" exitCode=0
Nov 03 16:10:55.802448 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:55.801988    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht" event={"ID":"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4","Type":"ContainerDied","Data":"2fc63eb85a4777d3fc73be548732ac7e7fd394e85fa66cfb390a1d54ee5e81b6"}
Nov 03 16:10:56.810752 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:56.810715    6506 generic.go:356] "Generic (PLEG): container finished" podID="b8136e22-5d4e-460b-b15e-defa2ca183e4" containerID="8bde5ef20b3c5b9b228338b29b6179aa93f8b9500331d43e79c97d6146012d8e" exitCode=0
Nov 03 16:10:56.811277 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:10:56.810865    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb" event={"ID":"b8136e22-5d4e-460b-b15e-defa2ca183e4","Type":"ContainerDied","Data":"8bde5ef20b3c5b9b228338b29b6179aa93f8b9500331d43e79c97d6146012d8e"}
Nov 03 16:10:58.817232 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:10:58.817091    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:11:04.878805 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:04.878208    6506 generic.go:356] "Generic (PLEG): container finished" podID="0704aaa3-8df1-41c9-8df3-0975a90ec44a" containerID="32a4304da3fdbd7507b1641340f5a176ab057dda72bffe15fa87cd8163777f54" exitCode=0
Nov 03 16:11:04.878805 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:04.878309    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn" event={"ID":"0704aaa3-8df1-41c9-8df3-0975a90ec44a","Type":"ContainerDied","Data":"32a4304da3fdbd7507b1641340f5a176ab057dda72bffe15fa87cd8163777f54"}
Nov 03 16:11:05.887812 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:05.887763    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6" event={"ID":"4c501915-1a3d-4309-bb99-54f5f3ec0294","Type":"ContainerStarted","Data":"c42f86b29470c1c2acd4c40cc38c08d52a8945aefd5e27a8a03a47331ae62862"}
Nov 03 16:11:05.913657 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:05.913606    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6" podStartSLOduration=2.384413668 podStartE2EDuration="31.913590446s" totalImagesPullingTime="29.529176778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:05.912957352 +0000 UTC m=+479.926355501" watchObservedRunningTime="2026-11-03 16:11:05.913590446 +0000 UTC m=+479.926988591"
Nov 03 16:11:06.078249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.078010    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"
Nov 03 16:11:06.185831 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.185773    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0704aaa3-8df1-41c9-8df3-0975a90ec44a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0704aaa3-8df1-41c9-8df3-0975a90ec44a-bundle\") pod \"0704aaa3-8df1-41c9-8df3-0975a90ec44a\" (UID: \"0704aaa3-8df1-41c9-8df3-0975a90ec44a\") "
Nov 03 16:11:06.186005 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.185869    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0704aaa3-8df1-41c9-8df3-0975a90ec44a-util\" (UniqueName: \"kubernetes.io/empty-dir/0704aaa3-8df1-41c9-8df3-0975a90ec44a-util\") pod \"0704aaa3-8df1-41c9-8df3-0975a90ec44a\" (UID: \"0704aaa3-8df1-41c9-8df3-0975a90ec44a\") "
Nov 03 16:11:06.186005 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.185917    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0704aaa3-8df1-41c9-8df3-0975a90ec44a-kube-api-access-ztpxt\" (UniqueName: \"kubernetes.io/projected/0704aaa3-8df1-41c9-8df3-0975a90ec44a-kube-api-access-ztpxt\") pod \"0704aaa3-8df1-41c9-8df3-0975a90ec44a\" (UID: \"0704aaa3-8df1-41c9-8df3-0975a90ec44a\") "
Nov 03 16:11:06.186420 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.186388    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0704aaa3-8df1-41c9-8df3-0975a90ec44a-bundle" pod "0704aaa3-8df1-41c9-8df3-0975a90ec44a" (UID: "0704aaa3-8df1-41c9-8df3-0975a90ec44a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:06.193329 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.193301    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0704aaa3-8df1-41c9-8df3-0975a90ec44a-util" pod "0704aaa3-8df1-41c9-8df3-0975a90ec44a" (UID: "0704aaa3-8df1-41c9-8df3-0975a90ec44a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:06.196333 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.196306    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0704aaa3-8df1-41c9-8df3-0975a90ec44a-kube-api-access-ztpxt" pod "0704aaa3-8df1-41c9-8df3-0975a90ec44a" (UID: "0704aaa3-8df1-41c9-8df3-0975a90ec44a"). InnerVolumeSpecName "kube-api-access-ztpxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:11:06.287058 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.287017    6506 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0704aaa3-8df1-41c9-8df3-0975a90ec44a-bundle\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:06.287058 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.287055    6506 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0704aaa3-8df1-41c9-8df3-0975a90ec44a-util\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:06.287253 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.287071    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztpxt\" (UniqueName: \"kubernetes.io/projected/0704aaa3-8df1-41c9-8df3-0975a90ec44a-kube-api-access-ztpxt\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:06.893706 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.893675    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn"
Nov 03 16:11:06.894196 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.893677    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168mdn" event={"ID":"0704aaa3-8df1-41c9-8df3-0975a90ec44a","Type":"ContainerDied","Data":"06b65e44c72eda6e838130f641e6167a7946cfd621a3f1c82fee7d912ec9fabf"}
Nov 03 16:11:06.894196 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.893830    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06b65e44c72eda6e838130f641e6167a7946cfd621a3f1c82fee7d912ec9fabf"
Nov 03 16:11:06.896034 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.896010    6506 generic.go:356] "Generic (PLEG): container finished" podID="4c501915-1a3d-4309-bb99-54f5f3ec0294" containerID="c42f86b29470c1c2acd4c40cc38c08d52a8945aefd5e27a8a03a47331ae62862" exitCode=0
Nov 03 16:11:06.896170 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.896076    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6" event={"ID":"4c501915-1a3d-4309-bb99-54f5f3ec0294","Type":"ContainerDied","Data":"c42f86b29470c1c2acd4c40cc38c08d52a8945aefd5e27a8a03a47331ae62862"}
Nov 03 16:11:06.898564 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.898533    6506 generic.go:356] "Generic (PLEG): container finished" podID="b8136e22-5d4e-460b-b15e-defa2ca183e4" containerID="03bb422b17bd13387297a508874789432797b1f8e69e721a5719c12d6148a087" exitCode=0
Nov 03 16:11:06.898564 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.898560    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb" event={"ID":"b8136e22-5d4e-460b-b15e-defa2ca183e4","Type":"ContainerDied","Data":"03bb422b17bd13387297a508874789432797b1f8e69e721a5719c12d6148a087"}
Nov 03 16:11:06.901062 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.901026    6506 generic.go:356] "Generic (PLEG): container finished" podID="44c8d92c-0b9e-42c7-9bcb-e626b371f4a4" containerID="bce4341f4e2cd33b4bbbbca0c5438a5f54d8f9b6b50a78594f461606881428ac" exitCode=0
Nov 03 16:11:06.901163 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:06.901095    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht" event={"ID":"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4","Type":"ContainerDied","Data":"bce4341f4e2cd33b4bbbbca0c5438a5f54d8f9b6b50a78594f461606881428ac"}
Nov 03 16:11:07.572722 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:11:07.572694    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8136e22_5d4e_460b_b15e_defa2ca183e4.slice/crio-conmon-03bb422b17bd13387297a508874789432797b1f8e69e721a5719c12d6148a087.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:11:07.906847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:07.906750    6506 generic.go:356] "Generic (PLEG): container finished" podID="5808de3e-b030-4290-98de-a2b476faac9b" containerID="1d57c864f0d56d5221b7fe806d909b924904e4a467b197670e06cd7ef45e1c7a" exitCode=0
Nov 03 16:11:07.907289 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:07.906837    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt" event={"ID":"5808de3e-b030-4290-98de-a2b476faac9b","Type":"ContainerDied","Data":"1d57c864f0d56d5221b7fe806d909b924904e4a467b197670e06cd7ef45e1c7a"}
Nov 03 16:11:07.908885 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:07.908862    6506 generic.go:356] "Generic (PLEG): container finished" podID="84f6338c-87ba-4910-b41a-e8d4f29b9d88" containerID="86b93dd4fbda1c4f8c888d03ef38a84b598d88fd0f4c88e84808d52d2dd57457" exitCode=0
Nov 03 16:11:07.908993 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:07.908927    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v" event={"ID":"84f6338c-87ba-4910-b41a-e8d4f29b9d88","Type":"ContainerDied","Data":"86b93dd4fbda1c4f8c888d03ef38a84b598d88fd0f4c88e84808d52d2dd57457"}
Nov 03 16:11:08.087042 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.086972    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"
Nov 03 16:11:08.109114 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.109085    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"
Nov 03 16:11:08.125339 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.125310    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"
Nov 03 16:11:08.206984 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.206953    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c501915-1a3d-4309-bb99-54f5f3ec0294-util\" (UniqueName: \"kubernetes.io/empty-dir/4c501915-1a3d-4309-bb99-54f5f3ec0294-util\") pod \"4c501915-1a3d-4309-bb99-54f5f3ec0294\" (UID: \"4c501915-1a3d-4309-bb99-54f5f3ec0294\") "
Nov 03 16:11:08.206984 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.206991    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-bundle\") pod \"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4\" (UID: \"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4\") "
Nov 03 16:11:08.207225 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.207011    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8136e22-5d4e-460b-b15e-defa2ca183e4-util\" (UniqueName: \"kubernetes.io/empty-dir/b8136e22-5d4e-460b-b15e-defa2ca183e4-util\") pod \"b8136e22-5d4e-460b-b15e-defa2ca183e4\" (UID: \"b8136e22-5d4e-460b-b15e-defa2ca183e4\") "
Nov 03 16:11:08.207225 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.207039    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-kube-api-access-nvfh7\" (UniqueName: \"kubernetes.io/projected/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-kube-api-access-nvfh7\") pod \"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4\" (UID: \"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4\") "
Nov 03 16:11:08.207225 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.207104    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8136e22-5d4e-460b-b15e-defa2ca183e4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8136e22-5d4e-460b-b15e-defa2ca183e4-bundle\") pod \"b8136e22-5d4e-460b-b15e-defa2ca183e4\" (UID: \"b8136e22-5d4e-460b-b15e-defa2ca183e4\") "
Nov 03 16:11:08.207225 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.207137    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8136e22-5d4e-460b-b15e-defa2ca183e4-kube-api-access-qzzjn\" (UniqueName: \"kubernetes.io/projected/b8136e22-5d4e-460b-b15e-defa2ca183e4-kube-api-access-qzzjn\") pod \"b8136e22-5d4e-460b-b15e-defa2ca183e4\" (UID: \"b8136e22-5d4e-460b-b15e-defa2ca183e4\") "
Nov 03 16:11:08.207225 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.207158    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c501915-1a3d-4309-bb99-54f5f3ec0294-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c501915-1a3d-4309-bb99-54f5f3ec0294-bundle\") pod \"4c501915-1a3d-4309-bb99-54f5f3ec0294\" (UID: \"4c501915-1a3d-4309-bb99-54f5f3ec0294\") "
Nov 03 16:11:08.207225 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.207191    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c501915-1a3d-4309-bb99-54f5f3ec0294-kube-api-access-pnxxd\" (UniqueName: \"kubernetes.io/projected/4c501915-1a3d-4309-bb99-54f5f3ec0294-kube-api-access-pnxxd\") pod \"4c501915-1a3d-4309-bb99-54f5f3ec0294\" (UID: \"4c501915-1a3d-4309-bb99-54f5f3ec0294\") "
Nov 03 16:11:08.207519 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.207262    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-util\" (UniqueName: \"kubernetes.io/empty-dir/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-util\") pod \"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4\" (UID: \"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4\") "
Nov 03 16:11:08.207894 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.207722    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8136e22-5d4e-460b-b15e-defa2ca183e4-bundle" pod "b8136e22-5d4e-460b-b15e-defa2ca183e4" (UID: "b8136e22-5d4e-460b-b15e-defa2ca183e4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:08.208081 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.208051    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c501915-1a3d-4309-bb99-54f5f3ec0294-bundle" pod "4c501915-1a3d-4309-bb99-54f5f3ec0294" (UID: "4c501915-1a3d-4309-bb99-54f5f3ec0294"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:08.208148 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.208096    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-bundle" pod "44c8d92c-0b9e-42c7-9bcb-e626b371f4a4" (UID: "44c8d92c-0b9e-42c7-9bcb-e626b371f4a4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:08.209636 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.209602    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8136e22-5d4e-460b-b15e-defa2ca183e4-kube-api-access-qzzjn" pod "b8136e22-5d4e-460b-b15e-defa2ca183e4" (UID: "b8136e22-5d4e-460b-b15e-defa2ca183e4"). InnerVolumeSpecName "kube-api-access-qzzjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:11:08.209636 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.209621    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-kube-api-access-nvfh7" pod "44c8d92c-0b9e-42c7-9bcb-e626b371f4a4" (UID: "44c8d92c-0b9e-42c7-9bcb-e626b371f4a4"). InnerVolumeSpecName "kube-api-access-nvfh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:11:08.210145 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.210121    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c501915-1a3d-4309-bb99-54f5f3ec0294-kube-api-access-pnxxd" pod "4c501915-1a3d-4309-bb99-54f5f3ec0294" (UID: "4c501915-1a3d-4309-bb99-54f5f3ec0294"). InnerVolumeSpecName "kube-api-access-pnxxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:11:08.214238 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.214219    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c501915-1a3d-4309-bb99-54f5f3ec0294-util" pod "4c501915-1a3d-4309-bb99-54f5f3ec0294" (UID: "4c501915-1a3d-4309-bb99-54f5f3ec0294"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:08.214324 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.214305    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-util" pod "44c8d92c-0b9e-42c7-9bcb-e626b371f4a4" (UID: "44c8d92c-0b9e-42c7-9bcb-e626b371f4a4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:08.214536 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.214517    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8136e22-5d4e-460b-b15e-defa2ca183e4-util" pod "b8136e22-5d4e-460b-b15e-defa2ca183e4" (UID: "b8136e22-5d4e-460b-b15e-defa2ca183e4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:08.308733 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.308696    6506 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-util\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:08.308733 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.308726    6506 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c501915-1a3d-4309-bb99-54f5f3ec0294-util\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:08.308733 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.308738    6506 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-bundle\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:08.308998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.308751    6506 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8136e22-5d4e-460b-b15e-defa2ca183e4-util\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:08.308998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.308765    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvfh7\" (UniqueName: \"kubernetes.io/projected/44c8d92c-0b9e-42c7-9bcb-e626b371f4a4-kube-api-access-nvfh7\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:08.308998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.308777    6506 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8136e22-5d4e-460b-b15e-defa2ca183e4-bundle\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:08.308998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.308788    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzzjn\" (UniqueName: \"kubernetes.io/projected/b8136e22-5d4e-460b-b15e-defa2ca183e4-kube-api-access-qzzjn\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:08.308998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.308826    6506 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c501915-1a3d-4309-bb99-54f5f3ec0294-bundle\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:08.308998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.308837    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnxxd\" (UniqueName: \"kubernetes.io/projected/4c501915-1a3d-4309-bb99-54f5f3ec0294-kube-api-access-pnxxd\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:08.769597 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.769565    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:08.914921 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.914882    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6" event={"ID":"4c501915-1a3d-4309-bb99-54f5f3ec0294","Type":"ContainerDied","Data":"0201549e2ef7a0f3d6b470b0d1eeff8fc853c2b256ea0cfc1e48a6f5ef8982c4"}
Nov 03 16:11:08.915316 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.914928    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0201549e2ef7a0f3d6b470b0d1eeff8fc853c2b256ea0cfc1e48a6f5ef8982c4"
Nov 03 16:11:08.915316 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.914918    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zvfd6"
Nov 03 16:11:08.916806 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.916771    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb"
Nov 03 16:11:08.916806 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.916779    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444qrkb" event={"ID":"b8136e22-5d4e-460b-b15e-defa2ca183e4","Type":"ContainerDied","Data":"99469a7cf4f92fc8786233dce8d3afc77c8bee9b9ea41315e687ed3b5b6f7207"}
Nov 03 16:11:08.916979 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.916839    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99469a7cf4f92fc8786233dce8d3afc77c8bee9b9ea41315e687ed3b5b6f7207"
Nov 03 16:11:08.919048 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.919026    6506 generic.go:356] "Generic (PLEG): container finished" podID="6756018e-938d-4549-b33c-d80c5eebd2a9" containerID="c4bf1f0d70c0e86412ec886af0f5267f070233cd97485018fa8850e337888b64" exitCode=0
Nov 03 16:11:08.919152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.919100    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx" event={"ID":"6756018e-938d-4549-b33c-d80c5eebd2a9","Type":"ContainerDied","Data":"c4bf1f0d70c0e86412ec886af0f5267f070233cd97485018fa8850e337888b64"}
Nov 03 16:11:08.926710 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.926686    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht"
Nov 03 16:11:08.926710 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.926696    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7tvht" event={"ID":"44c8d92c-0b9e-42c7-9bcb-e626b371f4a4","Type":"ContainerDied","Data":"7b15d57da11faf6eb6cdcf1e2240463685922f17c14a45bec992a4104ad5bb3b"}
Nov 03 16:11:08.926925 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:08.926761    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b15d57da11faf6eb6cdcf1e2240463685922f17c14a45bec992a4104ad5bb3b"
Nov 03 16:11:09.050596 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.050572    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"
Nov 03 16:11:09.074963 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.074930    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"
Nov 03 16:11:09.215968 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.215931    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f6338c-87ba-4910-b41a-e8d4f29b9d88-bundle\" (UniqueName: \"kubernetes.io/empty-dir/84f6338c-87ba-4910-b41a-e8d4f29b9d88-bundle\") pod \"84f6338c-87ba-4910-b41a-e8d4f29b9d88\" (UID: \"84f6338c-87ba-4910-b41a-e8d4f29b9d88\") "
Nov 03 16:11:09.216139 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.215978    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5808de3e-b030-4290-98de-a2b476faac9b-util\" (UniqueName: \"kubernetes.io/empty-dir/5808de3e-b030-4290-98de-a2b476faac9b-util\") pod \"5808de3e-b030-4290-98de-a2b476faac9b\" (UID: \"5808de3e-b030-4290-98de-a2b476faac9b\") "
Nov 03 16:11:09.216139 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.216046    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5808de3e-b030-4290-98de-a2b476faac9b-kube-api-access-hcmvz\" (UniqueName: \"kubernetes.io/projected/5808de3e-b030-4290-98de-a2b476faac9b-kube-api-access-hcmvz\") pod \"5808de3e-b030-4290-98de-a2b476faac9b\" (UID: \"5808de3e-b030-4290-98de-a2b476faac9b\") "
Nov 03 16:11:09.216139 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.216080    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5808de3e-b030-4290-98de-a2b476faac9b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5808de3e-b030-4290-98de-a2b476faac9b-bundle\") pod \"5808de3e-b030-4290-98de-a2b476faac9b\" (UID: \"5808de3e-b030-4290-98de-a2b476faac9b\") "
Nov 03 16:11:09.216139 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.216128    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f6338c-87ba-4910-b41a-e8d4f29b9d88-kube-api-access-sqc7h\" (UniqueName: \"kubernetes.io/projected/84f6338c-87ba-4910-b41a-e8d4f29b9d88-kube-api-access-sqc7h\") pod \"84f6338c-87ba-4910-b41a-e8d4f29b9d88\" (UID: \"84f6338c-87ba-4910-b41a-e8d4f29b9d88\") "
Nov 03 16:11:09.216360 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.216165    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f6338c-87ba-4910-b41a-e8d4f29b9d88-util\" (UniqueName: \"kubernetes.io/empty-dir/84f6338c-87ba-4910-b41a-e8d4f29b9d88-util\") pod \"84f6338c-87ba-4910-b41a-e8d4f29b9d88\" (UID: \"84f6338c-87ba-4910-b41a-e8d4f29b9d88\") "
Nov 03 16:11:09.216659 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.216630    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5808de3e-b030-4290-98de-a2b476faac9b-bundle" pod "5808de3e-b030-4290-98de-a2b476faac9b" (UID: "5808de3e-b030-4290-98de-a2b476faac9b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:09.216854 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.216826    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f6338c-87ba-4910-b41a-e8d4f29b9d88-bundle" pod "84f6338c-87ba-4910-b41a-e8d4f29b9d88" (UID: "84f6338c-87ba-4910-b41a-e8d4f29b9d88"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:09.218409 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.218378    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5808de3e-b030-4290-98de-a2b476faac9b-kube-api-access-hcmvz" pod "5808de3e-b030-4290-98de-a2b476faac9b" (UID: "5808de3e-b030-4290-98de-a2b476faac9b"). InnerVolumeSpecName "kube-api-access-hcmvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:11:09.218711 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.218693    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f6338c-87ba-4910-b41a-e8d4f29b9d88-kube-api-access-sqc7h" pod "84f6338c-87ba-4910-b41a-e8d4f29b9d88" (UID: "84f6338c-87ba-4910-b41a-e8d4f29b9d88"). InnerVolumeSpecName "kube-api-access-sqc7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:11:09.221907 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.221884    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5808de3e-b030-4290-98de-a2b476faac9b-util" pod "5808de3e-b030-4290-98de-a2b476faac9b" (UID: "5808de3e-b030-4290-98de-a2b476faac9b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:09.222914 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.222890    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f6338c-87ba-4910-b41a-e8d4f29b9d88-util" pod "84f6338c-87ba-4910-b41a-e8d4f29b9d88" (UID: "84f6338c-87ba-4910-b41a-e8d4f29b9d88"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:09.317370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.317273    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqc7h\" (UniqueName: \"kubernetes.io/projected/84f6338c-87ba-4910-b41a-e8d4f29b9d88-kube-api-access-sqc7h\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:09.317370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.317315    6506 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84f6338c-87ba-4910-b41a-e8d4f29b9d88-util\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:09.317370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.317329    6506 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84f6338c-87ba-4910-b41a-e8d4f29b9d88-bundle\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:09.317370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.317341    6506 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5808de3e-b030-4290-98de-a2b476faac9b-util\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:09.317370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.317352    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcmvz\" (UniqueName: \"kubernetes.io/projected/5808de3e-b030-4290-98de-a2b476faac9b-kube-api-access-hcmvz\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:09.317370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.317364    6506 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5808de3e-b030-4290-98de-a2b476faac9b-bundle\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:09.932348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.932320    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt"
Nov 03 16:11:09.932763 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.932323    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f7fgt" event={"ID":"5808de3e-b030-4290-98de-a2b476faac9b","Type":"ContainerDied","Data":"45e28d238f22555e878b65d30049a354579c6beff86f45d17a55a178c9bffe3b"}
Nov 03 16:11:09.932763 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.932447    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45e28d238f22555e878b65d30049a354579c6beff86f45d17a55a178c9bffe3b"
Nov 03 16:11:09.934373 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.934349    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v"
Nov 03 16:11:09.934485 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.934368    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aght7v" event={"ID":"84f6338c-87ba-4910-b41a-e8d4f29b9d88","Type":"ContainerDied","Data":"518e553007a8aa09356d3d641eef0b2743f72b5e258952f639d5af77eea243bb"}
Nov 03 16:11:09.934485 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:09.934404    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="518e553007a8aa09356d3d641eef0b2743f72b5e258952f639d5af77eea243bb"
Nov 03 16:11:10.063083 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:10.063056    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"
Nov 03 16:11:10.225933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:10.225845    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6756018e-938d-4549-b33c-d80c5eebd2a9-util\" (UniqueName: \"kubernetes.io/empty-dir/6756018e-938d-4549-b33c-d80c5eebd2a9-util\") pod \"6756018e-938d-4549-b33c-d80c5eebd2a9\" (UID: \"6756018e-938d-4549-b33c-d80c5eebd2a9\") "
Nov 03 16:11:10.225933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:10.225929    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6756018e-938d-4549-b33c-d80c5eebd2a9-kube-api-access-n6ksn\" (UniqueName: \"kubernetes.io/projected/6756018e-938d-4549-b33c-d80c5eebd2a9-kube-api-access-n6ksn\") pod \"6756018e-938d-4549-b33c-d80c5eebd2a9\" (UID: \"6756018e-938d-4549-b33c-d80c5eebd2a9\") "
Nov 03 16:11:10.226125 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:10.225948    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6756018e-938d-4549-b33c-d80c5eebd2a9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6756018e-938d-4549-b33c-d80c5eebd2a9-bundle\") pod \"6756018e-938d-4549-b33c-d80c5eebd2a9\" (UID: \"6756018e-938d-4549-b33c-d80c5eebd2a9\") "
Nov 03 16:11:10.226495 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:10.226464    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6756018e-938d-4549-b33c-d80c5eebd2a9-bundle" pod "6756018e-938d-4549-b33c-d80c5eebd2a9" (UID: "6756018e-938d-4549-b33c-d80c5eebd2a9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:10.228219 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:10.228189    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6756018e-938d-4549-b33c-d80c5eebd2a9-kube-api-access-n6ksn" pod "6756018e-938d-4549-b33c-d80c5eebd2a9" (UID: "6756018e-938d-4549-b33c-d80c5eebd2a9"). InnerVolumeSpecName "kube-api-access-n6ksn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:11:10.231103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:10.231084    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6756018e-938d-4549-b33c-d80c5eebd2a9-util" pod "6756018e-938d-4549-b33c-d80c5eebd2a9" (UID: "6756018e-938d-4549-b33c-d80c5eebd2a9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:10.327003 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:10.326966    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6ksn\" (UniqueName: \"kubernetes.io/projected/6756018e-938d-4549-b33c-d80c5eebd2a9-kube-api-access-n6ksn\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:10.327003 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:10.326994    6506 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6756018e-938d-4549-b33c-d80c5eebd2a9-bundle\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:10.327003 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:10.327003    6506 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6756018e-938d-4549-b33c-d80c5eebd2a9-util\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:10.940106 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:10.940069    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx" event={"ID":"6756018e-938d-4549-b33c-d80c5eebd2a9","Type":"ContainerDied","Data":"c81bf48bc1f8986c8d28e76faa5cce9f8f70db9a29d8fdacf2a3e14352e97574"}
Nov 03 16:11:10.940699 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:10.940116    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c81bf48bc1f8986c8d28e76faa5cce9f8f70db9a29d8fdacf2a3e14352e97574"
Nov 03 16:11:10.940699 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:10.940132    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvtlbx"
Nov 03 16:11:11.723028 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:11.722978    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:11.882467 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:11.882427    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:12.420399 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.420351    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:12.473289 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473257    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"]
Nov 03 16:11:12.473588 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473575    6506 cpu_manager.go:414] "containerMap: removing container" podUID="44c8d92c-0b9e-42c7-9bcb-e626b371f4a4" containerName="pull"
Nov 03 16:11:12.473630 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473590    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c8d92c-0b9e-42c7-9bcb-e626b371f4a4" containerName="pull"
Nov 03 16:11:12.473630 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473604    6506 cpu_manager.go:414] "containerMap: removing container" podUID="84f6338c-87ba-4910-b41a-e8d4f29b9d88" containerName="extract"
Nov 03 16:11:12.473630 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473609    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f6338c-87ba-4910-b41a-e8d4f29b9d88" containerName="extract"
Nov 03 16:11:12.473630 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473616    6506 cpu_manager.go:414] "containerMap: removing container" podUID="6756018e-938d-4549-b33c-d80c5eebd2a9" containerName="pull"
Nov 03 16:11:12.473630 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473620    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6756018e-938d-4549-b33c-d80c5eebd2a9" containerName="pull"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473633    6506 cpu_manager.go:414] "containerMap: removing container" podUID="0704aaa3-8df1-41c9-8df3-0975a90ec44a" containerName="pull"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473641    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0704aaa3-8df1-41c9-8df3-0975a90ec44a" containerName="pull"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473648    6506 cpu_manager.go:414] "containerMap: removing container" podUID="4c501915-1a3d-4309-bb99-54f5f3ec0294" containerName="extract"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473653    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c501915-1a3d-4309-bb99-54f5f3ec0294" containerName="extract"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473661    6506 cpu_manager.go:414] "containerMap: removing container" podUID="84f6338c-87ba-4910-b41a-e8d4f29b9d88" containerName="util"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473665    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f6338c-87ba-4910-b41a-e8d4f29b9d88" containerName="util"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473670    6506 cpu_manager.go:414] "containerMap: removing container" podUID="84f6338c-87ba-4910-b41a-e8d4f29b9d88" containerName="pull"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473674    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f6338c-87ba-4910-b41a-e8d4f29b9d88" containerName="pull"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473680    6506 cpu_manager.go:414] "containerMap: removing container" podUID="0704aaa3-8df1-41c9-8df3-0975a90ec44a" containerName="extract"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473684    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0704aaa3-8df1-41c9-8df3-0975a90ec44a" containerName="extract"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473690    6506 cpu_manager.go:414] "containerMap: removing container" podUID="b8136e22-5d4e-460b-b15e-defa2ca183e4" containerName="extract"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473693    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8136e22-5d4e-460b-b15e-defa2ca183e4" containerName="extract"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473700    6506 cpu_manager.go:414] "containerMap: removing container" podUID="44c8d92c-0b9e-42c7-9bcb-e626b371f4a4" containerName="util"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473706    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c8d92c-0b9e-42c7-9bcb-e626b371f4a4" containerName="util"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473721    6506 cpu_manager.go:414] "containerMap: removing container" podUID="6756018e-938d-4549-b33c-d80c5eebd2a9" containerName="util"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473727    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6756018e-938d-4549-b33c-d80c5eebd2a9" containerName="util"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473733    6506 cpu_manager.go:414] "containerMap: removing container" podUID="4c501915-1a3d-4309-bb99-54f5f3ec0294" containerName="pull"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473737    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c501915-1a3d-4309-bb99-54f5f3ec0294" containerName="pull"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473742    6506 cpu_manager.go:414] "containerMap: removing container" podUID="5808de3e-b030-4290-98de-a2b476faac9b" containerName="pull"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473746    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5808de3e-b030-4290-98de-a2b476faac9b" containerName="pull"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473751    6506 cpu_manager.go:414] "containerMap: removing container" podUID="44c8d92c-0b9e-42c7-9bcb-e626b371f4a4" containerName="extract"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473755    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c8d92c-0b9e-42c7-9bcb-e626b371f4a4" containerName="extract"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473761    6506 cpu_manager.go:414] "containerMap: removing container" podUID="b8136e22-5d4e-460b-b15e-defa2ca183e4" containerName="pull"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473765    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8136e22-5d4e-460b-b15e-defa2ca183e4" containerName="pull"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473773    6506 cpu_manager.go:414] "containerMap: removing container" podUID="5808de3e-b030-4290-98de-a2b476faac9b" containerName="util"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473777    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5808de3e-b030-4290-98de-a2b476faac9b" containerName="util"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473783    6506 cpu_manager.go:414] "containerMap: removing container" podUID="b8136e22-5d4e-460b-b15e-defa2ca183e4" containerName="util"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473787    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8136e22-5d4e-460b-b15e-defa2ca183e4" containerName="util"
Nov 03 16:11:12.473785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473813    6506 cpu_manager.go:414] "containerMap: removing container" podUID="6756018e-938d-4549-b33c-d80c5eebd2a9" containerName="extract"
Nov 03 16:11:12.474637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473818    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6756018e-938d-4549-b33c-d80c5eebd2a9" containerName="extract"
Nov 03 16:11:12.474637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473827    6506 cpu_manager.go:414] "containerMap: removing container" podUID="4c501915-1a3d-4309-bb99-54f5f3ec0294" containerName="util"
Nov 03 16:11:12.474637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473831    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c501915-1a3d-4309-bb99-54f5f3ec0294" containerName="util"
Nov 03 16:11:12.474637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473837    6506 cpu_manager.go:414] "containerMap: removing container" podUID="0704aaa3-8df1-41c9-8df3-0975a90ec44a" containerName="util"
Nov 03 16:11:12.474637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473841    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0704aaa3-8df1-41c9-8df3-0975a90ec44a" containerName="util"
Nov 03 16:11:12.474637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473848    6506 cpu_manager.go:414] "containerMap: removing container" podUID="5808de3e-b030-4290-98de-a2b476faac9b" containerName="extract"
Nov 03 16:11:12.474637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473852    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5808de3e-b030-4290-98de-a2b476faac9b" containerName="extract"
Nov 03 16:11:12.474637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473903    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f6338c-87ba-4910-b41a-e8d4f29b9d88" containerName="extract"
Nov 03 16:11:12.474637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473912    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c501915-1a3d-4309-bb99-54f5f3ec0294" containerName="extract"
Nov 03 16:11:12.474637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473918    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="0704aaa3-8df1-41c9-8df3-0975a90ec44a" containerName="extract"
Nov 03 16:11:12.474637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473922    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8136e22-5d4e-460b-b15e-defa2ca183e4" containerName="extract"
Nov 03 16:11:12.474637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473928    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c8d92c-0b9e-42c7-9bcb-e626b371f4a4" containerName="extract"
Nov 03 16:11:12.474637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473933    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="6756018e-938d-4549-b33c-d80c5eebd2a9" containerName="extract"
Nov 03 16:11:12.474637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.473938    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="5808de3e-b030-4290-98de-a2b476faac9b" containerName="extract"
Nov 03 16:11:12.477197 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.477181    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.491012 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.490990    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 16:11:12.491258 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.491239    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 16:11:12.491462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.491446    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-f626w\""
Nov 03 16:11:12.491513 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.491488    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 16:11:12.492034 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.492018    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 16:11:12.492102 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.492042    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 16:11:12.514408 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.514370    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"]
Nov 03 16:11:12.645066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.645026    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf23827c-bb93-4a99-a7ee-1686558b8a6d-manifests\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.645066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.645070    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmhnw\" (UniqueName: \"kubernetes.io/projected/cf23827c-bb93-4a99-a7ee-1686558b8a6d-kube-api-access-tmhnw\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.645289 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.645156    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf23827c-bb93-4a99-a7ee-1686558b8a6d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.645289 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.645205    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf23827c-bb93-4a99-a7ee-1686558b8a6d-tmp\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.645289 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.645235    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf23827c-bb93-4a99-a7ee-1686558b8a6d-webhook-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.645435 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.645292    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf23827c-bb93-4a99-a7ee-1686558b8a6d-metrics-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.693293 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.693206    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:12.745967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.745935    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf23827c-bb93-4a99-a7ee-1686558b8a6d-tmp\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.746108 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.745975    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf23827c-bb93-4a99-a7ee-1686558b8a6d-webhook-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.746108 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.745999    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf23827c-bb93-4a99-a7ee-1686558b8a6d-metrics-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.746108 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.746031    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf23827c-bb93-4a99-a7ee-1686558b8a6d-manifests\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.746108 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.746059    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmhnw\" (UniqueName: \"kubernetes.io/projected/cf23827c-bb93-4a99-a7ee-1686558b8a6d-kube-api-access-tmhnw\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.746108 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.746092    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf23827c-bb93-4a99-a7ee-1686558b8a6d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.746424 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.746396    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf23827c-bb93-4a99-a7ee-1686558b8a6d-manifests\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.748704 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.748678    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf23827c-bb93-4a99-a7ee-1686558b8a6d-tmp\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.748822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.748687    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf23827c-bb93-4a99-a7ee-1686558b8a6d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.748822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.748731    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf23827c-bb93-4a99-a7ee-1686558b8a6d-webhook-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.748938 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.748917    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf23827c-bb93-4a99-a7ee-1686558b8a6d-metrics-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.762637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.762612    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmhnw\" (UniqueName: \"kubernetes.io/projected/cf23827c-bb93-4a99-a7ee-1686558b8a6d-kube-api-access-tmhnw\") pod \"opendatahub-operator-controller-manager-d59cb86d5-fcmb5\" (UID: \"cf23827c-bb93-4a99-a7ee-1686558b8a6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.796508 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.796485    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:12.841848 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.841772    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:12.991565 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:12.991482    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:13.095219 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:13.094485    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"]
Nov 03 16:11:13.096749 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:11:13.096719    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf23827c_bb93_4a99_a7ee_1686558b8a6d.slice/crio-4b652dc9e81afef8d4416ed2c26d3b33909ec4ac306b58552bd04efb50952d98 WatchSource:0}: Error finding container 4b652dc9e81afef8d4416ed2c26d3b33909ec4ac306b58552bd04efb50952d98: Status 404 returned error can't find the container with id 4b652dc9e81afef8d4416ed2c26d3b33909ec4ac306b58552bd04efb50952d98
Nov 03 16:11:13.156812 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:13.156763    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:13.707833 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:13.707786    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:13.892325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:13.892288    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:13.957206 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:13.957168    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5" event={"ID":"cf23827c-bb93-4a99-a7ee-1686558b8a6d","Type":"ContainerStarted","Data":"4b652dc9e81afef8d4416ed2c26d3b33909ec4ac306b58552bd04efb50952d98"}
Nov 03 16:11:15.133846 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:15.133814    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:15.292514 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:15.292471    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:16.716502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:16.716473    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:16.971534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:16.971504    6506 generic.go:356] "Generic (PLEG): container finished" podID="cf23827c-bb93-4a99-a7ee-1686558b8a6d" containerID="87a68e50563441267f2cfa933f55bffbf58d6c0fd28f3ea001a551a641ffa165" exitCode=0
Nov 03 16:11:16.971708 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:16.971543    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5" event={"ID":"cf23827c-bb93-4a99-a7ee-1686558b8a6d","Type":"ContainerDied","Data":"87a68e50563441267f2cfa933f55bffbf58d6c0fd28f3ea001a551a641ffa165"}
Nov 03 16:11:17.607693 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:11:17.607659    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8136e22_5d4e_460b_b15e_defa2ca183e4.slice/crio-conmon-03bb422b17bd13387297a508874789432797b1f8e69e721a5719c12d6148a087.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:11:17.977704 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:17.977668    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5" event={"ID":"cf23827c-bb93-4a99-a7ee-1686558b8a6d","Type":"ContainerStarted","Data":"14bc8866871db8aa1770170d387b5fe676440587c2eaa7cb3dff75811482ae34"}
Nov 03 16:11:17.978098 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:17.977843    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:18.070058 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:18.070005    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5" podStartSLOduration=2.279596721 podStartE2EDuration="6.069986956s" totalImagesPullingTime="3.338955181s" totalInitContainerRuntime="451.435054ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:18.067663211 +0000 UTC m=+492.081061357" watchObservedRunningTime="2026-11-03 16:11:18.069986956 +0000 UTC m=+492.083385101"
Nov 03 16:11:18.430356 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:18.430321    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:18.654194 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:18.654160    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:19.769223 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:11:19.769183    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8136e22_5d4e_460b_b15e_defa2ca183e4.slice/crio-conmon-03bb422b17bd13387297a508874789432797b1f8e69e721a5719c12d6148a087.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:11:21.080479 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.080445    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9"]
Nov 03 16:11:21.083849 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.083832    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9"
Nov 03 16:11:21.093493 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.093464    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-skznr\""
Nov 03 16:11:21.093625 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.093577    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 16:11:21.093691 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.093628    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:11:21.100903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.100876    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9"]
Nov 03 16:11:21.222873 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.222840    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vpsr\" (UniqueName: \"kubernetes.io/projected/3ca6b478-40da-4f6d-aa5e-5dce03e13d26-kube-api-access-4vpsr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9\" (UID: \"3ca6b478-40da-4f6d-aa5e-5dce03e13d26\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9"
Nov 03 16:11:21.223003 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.222921    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca6b478-40da-4f6d-aa5e-5dce03e13d26-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9\" (UID: \"3ca6b478-40da-4f6d-aa5e-5dce03e13d26\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9"
Nov 03 16:11:21.324172 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.324142    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vpsr\" (UniqueName: \"kubernetes.io/projected/3ca6b478-40da-4f6d-aa5e-5dce03e13d26-kube-api-access-4vpsr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9\" (UID: \"3ca6b478-40da-4f6d-aa5e-5dce03e13d26\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9"
Nov 03 16:11:21.324172 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.324177    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca6b478-40da-4f6d-aa5e-5dce03e13d26-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9\" (UID: \"3ca6b478-40da-4f6d-aa5e-5dce03e13d26\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9"
Nov 03 16:11:21.324513 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.324495    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca6b478-40da-4f6d-aa5e-5dce03e13d26-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9\" (UID: \"3ca6b478-40da-4f6d-aa5e-5dce03e13d26\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9"
Nov 03 16:11:21.337094 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.337031    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vpsr\" (UniqueName: \"kubernetes.io/projected/3ca6b478-40da-4f6d-aa5e-5dce03e13d26-kube-api-access-4vpsr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9\" (UID: \"3ca6b478-40da-4f6d-aa5e-5dce03e13d26\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9"
Nov 03 16:11:21.400505 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.400477    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9"
Nov 03 16:11:21.564425 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.564389    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-scbnj"]
Nov 03 16:11:21.567924 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.567900    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-scbnj"
Nov 03 16:11:21.572655 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.572632    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:11:21.572768 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.572676    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 16:11:21.580037 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.580018    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-scbnj"]
Nov 03 16:11:21.608705 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.608540    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-7c2nl\""
Nov 03 16:11:21.677871 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.677851    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9"]
Nov 03 16:11:21.680435 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:11:21.680409    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca6b478_40da_4f6d_aa5e_5dce03e13d26.slice/crio-446f9cd750a9c4fb4e7e6fcb7ac8333668bed1c93448106f045f9d33f640c0e5 WatchSource:0}: Error finding container 446f9cd750a9c4fb4e7e6fcb7ac8333668bed1c93448106f045f9d33f640c0e5: Status 404 returned error can't find the container with id 446f9cd750a9c4fb4e7e6fcb7ac8333668bed1c93448106f045f9d33f640c0e5
Nov 03 16:11:21.729321 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.729297    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjq94\" (UniqueName: \"kubernetes.io/projected/7849bd02-4d3c-45c7-8415-78200374eba7-kube-api-access-qjq94\") pod \"jobset-operator-674cd59dc5-scbnj\" (UID: \"7849bd02-4d3c-45c7-8415-78200374eba7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-scbnj"
Nov 03 16:11:21.729431 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.729373    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7849bd02-4d3c-45c7-8415-78200374eba7-tmp\") pod \"jobset-operator-674cd59dc5-scbnj\" (UID: \"7849bd02-4d3c-45c7-8415-78200374eba7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-scbnj"
Nov 03 16:11:21.830628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.830600    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7849bd02-4d3c-45c7-8415-78200374eba7-tmp\") pod \"jobset-operator-674cd59dc5-scbnj\" (UID: \"7849bd02-4d3c-45c7-8415-78200374eba7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-scbnj"
Nov 03 16:11:21.830734 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.830657    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjq94\" (UniqueName: \"kubernetes.io/projected/7849bd02-4d3c-45c7-8415-78200374eba7-kube-api-access-qjq94\") pod \"jobset-operator-674cd59dc5-scbnj\" (UID: \"7849bd02-4d3c-45c7-8415-78200374eba7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-scbnj"
Nov 03 16:11:21.830988 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.830972    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7849bd02-4d3c-45c7-8415-78200374eba7-tmp\") pod \"jobset-operator-674cd59dc5-scbnj\" (UID: \"7849bd02-4d3c-45c7-8415-78200374eba7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-scbnj"
Nov 03 16:11:21.849677 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.849654    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjq94\" (UniqueName: \"kubernetes.io/projected/7849bd02-4d3c-45c7-8415-78200374eba7-kube-api-access-qjq94\") pod \"jobset-operator-674cd59dc5-scbnj\" (UID: \"7849bd02-4d3c-45c7-8415-78200374eba7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-scbnj"
Nov 03 16:11:21.892417 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.892367    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-scbnj"
Nov 03 16:11:21.995907 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:21.995876    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9" event={"ID":"3ca6b478-40da-4f6d-aa5e-5dce03e13d26","Type":"ContainerStarted","Data":"446f9cd750a9c4fb4e7e6fcb7ac8333668bed1c93448106f045f9d33f640c0e5"}
Nov 03 16:11:22.168638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:22.168611    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-scbnj"]
Nov 03 16:11:22.170968 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:11:22.170942    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7849bd02_4d3c_45c7_8415_78200374eba7.slice/crio-ff2e5607291a4b8b004d377e884efc312b6886647b2452246b8b01c9688909a3 WatchSource:0}: Error finding container ff2e5607291a4b8b004d377e884efc312b6886647b2452246b8b01c9688909a3: Status 404 returned error can't find the container with id ff2e5607291a4b8b004d377e884efc312b6886647b2452246b8b01c9688909a3
Nov 03 16:11:22.370337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:22.370309    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:23.001416 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:23.001384    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-scbnj" event={"ID":"7849bd02-4d3c-45c7-8415-78200374eba7","Type":"ContainerStarted","Data":"ff2e5607291a4b8b004d377e884efc312b6886647b2452246b8b01c9688909a3"}
Nov 03 16:11:23.315187 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:23.315153    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:23.473208 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:23.473174    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:26.019401 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.018944    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9" event={"ID":"3ca6b478-40da-4f6d-aa5e-5dce03e13d26","Type":"ContainerStarted","Data":"4487566b017223c72cd911871e9335b63430e599742d19ddb495d1a3dd5f3b8f"}
Nov 03 16:11:26.047258 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.047209    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rdgg9" podStartSLOduration=1.552034671 podStartE2EDuration="5.047191027s" totalImagesPullingTime="3.495156356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:26.043467735 +0000 UTC m=+500.056865884" watchObservedRunningTime="2026-11-03 16:11:26.047191027 +0000 UTC m=+500.060589171"
Nov 03 16:11:26.056096 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.056063    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:26.108407 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.108378    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r"]
Nov 03 16:11:26.112152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.112126    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r"
Nov 03 16:11:26.122958 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.122936    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 16:11:26.122958 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.122936    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-z7cvh\""
Nov 03 16:11:26.123106 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.123000    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 16:11:26.124232 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.124209    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r"]
Nov 03 16:11:26.167310 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.167273    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5rcr\" (UniqueName: \"kubernetes.io/projected/b017bcd4-4c3b-489d-9f98-3d8840edda8e-kube-api-access-t5rcr\") pod \"cert-manager-webhook-5dbbbf75cc-2m69r\" (UID: \"b017bcd4-4c3b-489d-9f98-3d8840edda8e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r"
Nov 03 16:11:26.167449 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.167325    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b017bcd4-4c3b-489d-9f98-3d8840edda8e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2m69r\" (UID: \"b017bcd4-4c3b-489d-9f98-3d8840edda8e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r"
Nov 03 16:11:26.226158 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.226133    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:26.268196 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.268167    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rcr\" (UniqueName: \"kubernetes.io/projected/b017bcd4-4c3b-489d-9f98-3d8840edda8e-kube-api-access-t5rcr\") pod \"cert-manager-webhook-5dbbbf75cc-2m69r\" (UID: \"b017bcd4-4c3b-489d-9f98-3d8840edda8e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r"
Nov 03 16:11:26.268321 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.268225    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b017bcd4-4c3b-489d-9f98-3d8840edda8e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2m69r\" (UID: \"b017bcd4-4c3b-489d-9f98-3d8840edda8e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r"
Nov 03 16:11:26.280656 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.280592    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b017bcd4-4c3b-489d-9f98-3d8840edda8e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2m69r\" (UID: \"b017bcd4-4c3b-489d-9f98-3d8840edda8e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r"
Nov 03 16:11:26.280809 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.280775    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5rcr\" (UniqueName: \"kubernetes.io/projected/b017bcd4-4c3b-489d-9f98-3d8840edda8e-kube-api-access-t5rcr\") pod \"cert-manager-webhook-5dbbbf75cc-2m69r\" (UID: \"b017bcd4-4c3b-489d-9f98-3d8840edda8e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r"
Nov 03 16:11:26.424899 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.424871    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r"
Nov 03 16:11:26.745336 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:26.744930    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r"]
Nov 03 16:11:26.749739 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:11:26.748739    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb017bcd4_4c3b_489d_9f98_3d8840edda8e.slice/crio-a9c87a733af580c98a5ecb1a9be14936b017c29218bed196805ec64627b272da WatchSource:0}: Error finding container a9c87a733af580c98a5ecb1a9be14936b017c29218bed196805ec64627b272da: Status 404 returned error can't find the container with id a9c87a733af580c98a5ecb1a9be14936b017c29218bed196805ec64627b272da
Nov 03 16:11:27.023960 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:27.023923    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r" event={"ID":"b017bcd4-4c3b-489d-9f98-3d8840edda8e","Type":"ContainerStarted","Data":"a9c87a733af580c98a5ecb1a9be14936b017c29218bed196805ec64627b272da"}
Nov 03 16:11:27.025575 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:27.025550    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-scbnj" event={"ID":"7849bd02-4d3c-45c7-8415-78200374eba7","Type":"ContainerStarted","Data":"7228f358d9db487d4db948764690e396b36c725c127ad462c9ce0004ee960964"}
Nov 03 16:11:27.048261 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:27.048204    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-scbnj" podStartSLOduration=2.029182172 podStartE2EDuration="6.04819122s" totalImagesPullingTime="4.019009048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:27.0480346 +0000 UTC m=+501.061432748" watchObservedRunningTime="2026-11-03 16:11:27.04819122 +0000 UTC m=+501.061589365"
Nov 03 16:11:27.655622 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:11:27.655588    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8136e22_5d4e_460b_b15e_defa2ca183e4.slice/crio-conmon-03bb422b17bd13387297a508874789432797b1f8e69e721a5719c12d6148a087.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:11:27.816837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:27.816778    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:28.385352 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:28.385293    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:28.800547 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:11:28.800496    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:11:28.984895 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:28.984863    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-fcmb5"
Nov 03 16:11:30.042370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:30.042286    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r" event={"ID":"b017bcd4-4c3b-489d-9f98-3d8840edda8e","Type":"ContainerStarted","Data":"47953609c4dc6e4c6e07fbc72f7463d62fa9b207e50fb2da6331e76aaa0b76e7"}
Nov 03 16:11:30.042749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:30.042382    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r"
Nov 03 16:11:30.063069 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:30.063018    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r" podStartSLOduration=1.021970741 podStartE2EDuration="4.063003354s" totalImagesPullingTime="3.041032613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:30.062927584 +0000 UTC m=+504.076325730" watchObservedRunningTime="2026-11-03 16:11:30.063003354 +0000 UTC m=+504.076401494"
Nov 03 16:11:32.094474 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:32.094436    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:33.120149 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:33.120115    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:33.279156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:33.279125    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:34.800338 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:11:34.800297    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8136e22_5d4e_460b_b15e_defa2ca183e4.slice/crio-conmon-03bb422b17bd13387297a508874789432797b1f8e69e721a5719c12d6148a087.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:11:35.589217 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:11:35.589178    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8136e22_5d4e_460b_b15e_defa2ca183e4.slice/crio-conmon-03bb422b17bd13387297a508874789432797b1f8e69e721a5719c12d6148a087.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:11:35.589410 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:11:35.589385    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8136e22_5d4e_460b_b15e_defa2ca183e4.slice/crio-conmon-03bb422b17bd13387297a508874789432797b1f8e69e721a5719c12d6148a087.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:11:36.049509 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:36.049478    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r"
Nov 03 16:11:36.704939 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:36.704903    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:37.244582 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.244546    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:37.666275 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:11:37.666245    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8136e22_5d4e_460b_b15e_defa2ca183e4.slice/crio-conmon-03bb422b17bd13387297a508874789432797b1f8e69e721a5719c12d6148a087.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:11:37.748285 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.748251    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-98cxp"]
Nov 03 16:11:37.750752 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.750735    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98cxp"
Nov 03 16:11:37.759617 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.759598    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-jqvh8\""
Nov 03 16:11:37.764470 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.764451    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-98cxp"]
Nov 03 16:11:37.797658 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.797633    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-jh5jf"]
Nov 03 16:11:37.799590 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.799575    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jh5jf"
Nov 03 16:11:37.811660 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.811636    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jh5jf"]
Nov 03 16:11:37.859160 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.859134    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-4v67t\""
Nov 03 16:11:37.860398 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.860381    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97653b5e-1bd5-4b67-aac0-79dfc0cc274a-bound-sa-token\") pod \"cert-manager-597b95cb84-jh5jf\" (UID: \"97653b5e-1bd5-4b67-aac0-79dfc0cc274a\") " pod="cert-manager/cert-manager-597b95cb84-jh5jf"
Nov 03 16:11:37.860453 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.860414    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwzf4\" (UniqueName: \"kubernetes.io/projected/97653b5e-1bd5-4b67-aac0-79dfc0cc274a-kube-api-access-bwzf4\") pod \"cert-manager-597b95cb84-jh5jf\" (UID: \"97653b5e-1bd5-4b67-aac0-79dfc0cc274a\") " pod="cert-manager/cert-manager-597b95cb84-jh5jf"
Nov 03 16:11:37.860453 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.860440    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf9b5\" (UniqueName: \"kubernetes.io/projected/8f7520e3-cae5-4101-9890-0579ed0ede69-kube-api-access-gf9b5\") pod \"cert-manager-cainjector-5c57bbd4fc-98cxp\" (UID: \"8f7520e3-cae5-4101-9890-0579ed0ede69\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98cxp"
Nov 03 16:11:37.860605 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.860586    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f7520e3-cae5-4101-9890-0579ed0ede69-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-98cxp\" (UID: \"8f7520e3-cae5-4101-9890-0579ed0ede69\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98cxp"
Nov 03 16:11:37.961782 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.961694    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97653b5e-1bd5-4b67-aac0-79dfc0cc274a-bound-sa-token\") pod \"cert-manager-597b95cb84-jh5jf\" (UID: \"97653b5e-1bd5-4b67-aac0-79dfc0cc274a\") " pod="cert-manager/cert-manager-597b95cb84-jh5jf"
Nov 03 16:11:37.961782 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.961731    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwzf4\" (UniqueName: \"kubernetes.io/projected/97653b5e-1bd5-4b67-aac0-79dfc0cc274a-kube-api-access-bwzf4\") pod \"cert-manager-597b95cb84-jh5jf\" (UID: \"97653b5e-1bd5-4b67-aac0-79dfc0cc274a\") " pod="cert-manager/cert-manager-597b95cb84-jh5jf"
Nov 03 16:11:37.961782 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.961754    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf9b5\" (UniqueName: \"kubernetes.io/projected/8f7520e3-cae5-4101-9890-0579ed0ede69-kube-api-access-gf9b5\") pod \"cert-manager-cainjector-5c57bbd4fc-98cxp\" (UID: \"8f7520e3-cae5-4101-9890-0579ed0ede69\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98cxp"
Nov 03 16:11:37.962067 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.961817    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f7520e3-cae5-4101-9890-0579ed0ede69-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-98cxp\" (UID: \"8f7520e3-cae5-4101-9890-0579ed0ede69\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98cxp"
Nov 03 16:11:37.976344 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.976316    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf9b5\" (UniqueName: \"kubernetes.io/projected/8f7520e3-cae5-4101-9890-0579ed0ede69-kube-api-access-gf9b5\") pod \"cert-manager-cainjector-5c57bbd4fc-98cxp\" (UID: \"8f7520e3-cae5-4101-9890-0579ed0ede69\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98cxp"
Nov 03 16:11:37.976886 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.976868    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f7520e3-cae5-4101-9890-0579ed0ede69-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-98cxp\" (UID: \"8f7520e3-cae5-4101-9890-0579ed0ede69\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98cxp"
Nov 03 16:11:37.977229 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.977211    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97653b5e-1bd5-4b67-aac0-79dfc0cc274a-bound-sa-token\") pod \"cert-manager-597b95cb84-jh5jf\" (UID: \"97653b5e-1bd5-4b67-aac0-79dfc0cc274a\") " pod="cert-manager/cert-manager-597b95cb84-jh5jf"
Nov 03 16:11:37.977344 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:37.977324    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwzf4\" (UniqueName: \"kubernetes.io/projected/97653b5e-1bd5-4b67-aac0-79dfc0cc274a-kube-api-access-bwzf4\") pod \"cert-manager-597b95cb84-jh5jf\" (UID: \"97653b5e-1bd5-4b67-aac0-79dfc0cc274a\") " pod="cert-manager/cert-manager-597b95cb84-jh5jf"
Nov 03 16:11:38.061197 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:38.061160    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98cxp"
Nov 03 16:11:38.109096 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:38.109061    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jh5jf"
Nov 03 16:11:38.333711 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:38.333637    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-98cxp"]
Nov 03 16:11:38.336302 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:11:38.336273    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f7520e3_cae5_4101_9890_0579ed0ede69.slice/crio-050faf995a062098baf7be66ae78c89cc212d4bca9d0bc6f57cc16a66bfbc9df WatchSource:0}: Error finding container 050faf995a062098baf7be66ae78c89cc212d4bca9d0bc6f57cc16a66bfbc9df: Status 404 returned error can't find the container with id 050faf995a062098baf7be66ae78c89cc212d4bca9d0bc6f57cc16a66bfbc9df
Nov 03 16:11:38.390405 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:38.390382    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jh5jf"]
Nov 03 16:11:38.393161 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:11:38.393137    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97653b5e_1bd5_4b67_aac0_79dfc0cc274a.slice/crio-38e9a2b406846439442f89cff43f6177400351735d2158533d1d381466f53c5c WatchSource:0}: Error finding container 38e9a2b406846439442f89cff43f6177400351735d2158533d1d381466f53c5c: Status 404 returned error can't find the container with id 38e9a2b406846439442f89cff43f6177400351735d2158533d1d381466f53c5c
Nov 03 16:11:39.086178 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:39.086146    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jh5jf" event={"ID":"97653b5e-1bd5-4b67-aac0-79dfc0cc274a","Type":"ContainerStarted","Data":"555055d634f18c0e39964809829c9f5d9ad86686f50856a6c424688a3b0e8d55"}
Nov 03 16:11:39.086367 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:39.086190    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jh5jf" event={"ID":"97653b5e-1bd5-4b67-aac0-79dfc0cc274a","Type":"ContainerStarted","Data":"38e9a2b406846439442f89cff43f6177400351735d2158533d1d381466f53c5c"}
Nov 03 16:11:39.087886 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:39.087853    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98cxp" event={"ID":"8f7520e3-cae5-4101-9890-0579ed0ede69","Type":"ContainerStarted","Data":"9a6551a41581ebe628884ef79a538a293eaf71e93289c76074b5b309c2d8be31"}
Nov 03 16:11:39.087984 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:39.087890    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98cxp" event={"ID":"8f7520e3-cae5-4101-9890-0579ed0ede69","Type":"ContainerStarted","Data":"050faf995a062098baf7be66ae78c89cc212d4bca9d0bc6f57cc16a66bfbc9df"}
Nov 03 16:11:39.106168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:39.106122    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-jh5jf" podStartSLOduration=2.106107678 podStartE2EDuration="2.106107678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:39.105864154 +0000 UTC m=+513.119262299" watchObservedRunningTime="2026-11-03 16:11:39.106107678 +0000 UTC m=+513.119505824"
Nov 03 16:11:39.135421 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:39.135379    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98cxp" podStartSLOduration=2.135367751 podStartE2EDuration="2.135367751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:39.134963773 +0000 UTC m=+513.148361919" watchObservedRunningTime="2026-11-03 16:11:39.135367751 +0000 UTC m=+513.148765897"
Nov 03 16:11:39.306620 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:39.306592    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:39.480639 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:39.480563    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:39.725132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:39.725105    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:40.510304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:40.510215    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:43.422428 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:43.422392    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:43.582585 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:43.582550    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:43.738107 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:43.738010    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:44.506357 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:44.506324    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:44.916295 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:44.916260    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:45.176939 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:45.176864    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:46.345899 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:46.345871    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:47.410220 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:47.410190    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:47.716490 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:11:47.716401    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8136e22_5d4e_460b_b15e_defa2ca183e4.slice/crio-conmon-03bb422b17bd13387297a508874789432797b1f8e69e721a5719c12d6148a087.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:11:47.720541 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:47.720463    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:49.558025 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:49.557989    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:49.721818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:49.721769    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:49.769171 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:11:49.769138    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8136e22_5d4e_460b_b15e_defa2ca183e4.slice/crio-conmon-03bb422b17bd13387297a508874789432797b1f8e69e721a5719c12d6148a087.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:11:52.320625 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:52.320591    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:52.487341 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:52.487306    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:54.376221 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.376186    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:54.674925 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.674843    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq"]
Nov 03 16:11:54.681214 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.681187    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq"
Nov 03 16:11:54.694305 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.694277    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq"]
Nov 03 16:11:54.713711 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.713687    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 16:11:54.714619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.714598    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 16:11:54.714744 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.714683    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-z9774\""
Nov 03 16:11:54.767031 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.766997    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:54.814683 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.814649    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbeb1f8a-f73e-4b82-bdc5-048c3e57453e-tls-certs\") pod \"kube-auth-proxy-864dbfdb86-4kdzq\" (UID: \"cbeb1f8a-f73e-4b82-bdc5-048c3e57453e\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq"
Nov 03 16:11:54.814683 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.814686    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbeb1f8a-f73e-4b82-bdc5-048c3e57453e-tmp\") pod \"kube-auth-proxy-864dbfdb86-4kdzq\" (UID: \"cbeb1f8a-f73e-4b82-bdc5-048c3e57453e\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq"
Nov 03 16:11:54.814890 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.814723    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thgc9\" (UniqueName: \"kubernetes.io/projected/cbeb1f8a-f73e-4b82-bdc5-048c3e57453e-kube-api-access-thgc9\") pod \"kube-auth-proxy-864dbfdb86-4kdzq\" (UID: \"cbeb1f8a-f73e-4b82-bdc5-048c3e57453e\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq"
Nov 03 16:11:54.915728 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.915687    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbeb1f8a-f73e-4b82-bdc5-048c3e57453e-tls-certs\") pod \"kube-auth-proxy-864dbfdb86-4kdzq\" (UID: \"cbeb1f8a-f73e-4b82-bdc5-048c3e57453e\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq"
Nov 03 16:11:54.915893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.915734    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbeb1f8a-f73e-4b82-bdc5-048c3e57453e-tmp\") pod \"kube-auth-proxy-864dbfdb86-4kdzq\" (UID: \"cbeb1f8a-f73e-4b82-bdc5-048c3e57453e\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq"
Nov 03 16:11:54.915893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.915781    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thgc9\" (UniqueName: \"kubernetes.io/projected/cbeb1f8a-f73e-4b82-bdc5-048c3e57453e-kube-api-access-thgc9\") pod \"kube-auth-proxy-864dbfdb86-4kdzq\" (UID: \"cbeb1f8a-f73e-4b82-bdc5-048c3e57453e\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq"
Nov 03 16:11:54.918297 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.918265    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbeb1f8a-f73e-4b82-bdc5-048c3e57453e-tmp\") pod \"kube-auth-proxy-864dbfdb86-4kdzq\" (UID: \"cbeb1f8a-f73e-4b82-bdc5-048c3e57453e\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq"
Nov 03 16:11:54.918566 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.918548    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbeb1f8a-f73e-4b82-bdc5-048c3e57453e-tls-certs\") pod \"kube-auth-proxy-864dbfdb86-4kdzq\" (UID: \"cbeb1f8a-f73e-4b82-bdc5-048c3e57453e\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq"
Nov 03 16:11:54.932289 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.932229    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thgc9\" (UniqueName: \"kubernetes.io/projected/cbeb1f8a-f73e-4b82-bdc5-048c3e57453e-kube-api-access-thgc9\") pod \"kube-auth-proxy-864dbfdb86-4kdzq\" (UID: \"cbeb1f8a-f73e-4b82-bdc5-048c3e57453e\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq"
Nov 03 16:11:54.957096 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.957064    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:54.996123 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:54.996089    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq"
Nov 03 16:11:55.124703 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:55.124659    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:55.283812 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:55.283776    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq"]
Nov 03 16:11:55.286418 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:11:55.286392    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbeb1f8a_f73e_4b82_bdc5_048c3e57453e.slice/crio-dbadbdd38c863caad2f95f5ec89ba192f8bff498383c814de944e2ce98fadf58 WatchSource:0}: Error finding container dbadbdd38c863caad2f95f5ec89ba192f8bff498383c814de944e2ce98fadf58: Status 404 returned error can't find the container with id dbadbdd38c863caad2f95f5ec89ba192f8bff498383c814de944e2ce98fadf58
Nov 03 16:11:55.804842 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:55.804784    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:56.030648 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:56.030560    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:56.165318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:56.165229    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq" event={"ID":"cbeb1f8a-f73e-4b82-bdc5-048c3e57453e","Type":"ContainerStarted","Data":"dbadbdd38c863caad2f95f5ec89ba192f8bff498383c814de944e2ce98fadf58"}
Nov 03 16:11:56.384766 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:56.384730    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:57.238502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.238470    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:57.396830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.396774    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:57.572884 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.572849    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:11:57.612153 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.612118    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"]
Nov 03 16:11:57.615999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.615977    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:11:57.628746 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.628710    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 03 16:11:57.629875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.629852    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-crwj2\""
Nov 03 16:11:57.630002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.629876    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 03 16:11:57.630002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.629937    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 03 16:11:57.630410 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.630382    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"]
Nov 03 16:11:57.750223 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.750189    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be2806bc-640c-4904-8f88-8ccc2d58c6ae-metrics-certs\") pod \"jobset-controller-manager-6fb6bc9cbf-rbvsh\" (UID: \"be2806bc-640c-4904-8f88-8ccc2d58c6ae\") " pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:11:57.750441 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.750280    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/be2806bc-640c-4904-8f88-8ccc2d58c6ae-manager-config\") pod \"jobset-controller-manager-6fb6bc9cbf-rbvsh\" (UID: \"be2806bc-640c-4904-8f88-8ccc2d58c6ae\") " pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:11:57.750441 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.750391    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be2806bc-640c-4904-8f88-8ccc2d58c6ae-cert\") pod \"jobset-controller-manager-6fb6bc9cbf-rbvsh\" (UID: \"be2806bc-640c-4904-8f88-8ccc2d58c6ae\") " pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:11:57.750597 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.750469    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zjr2\" (UniqueName: \"kubernetes.io/projected/be2806bc-640c-4904-8f88-8ccc2d58c6ae-kube-api-access-6zjr2\") pod \"jobset-controller-manager-6fb6bc9cbf-rbvsh\" (UID: \"be2806bc-640c-4904-8f88-8ccc2d58c6ae\") " pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:11:57.764142 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:11:57.764118    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8136e22_5d4e_460b_b15e_defa2ca183e4.slice/crio-conmon-03bb422b17bd13387297a508874789432797b1f8e69e721a5719c12d6148a087.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:11:57.851202 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.851174    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be2806bc-640c-4904-8f88-8ccc2d58c6ae-cert\") pod \"jobset-controller-manager-6fb6bc9cbf-rbvsh\" (UID: \"be2806bc-640c-4904-8f88-8ccc2d58c6ae\") " pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:11:57.851329 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.851207    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zjr2\" (UniqueName: \"kubernetes.io/projected/be2806bc-640c-4904-8f88-8ccc2d58c6ae-kube-api-access-6zjr2\") pod \"jobset-controller-manager-6fb6bc9cbf-rbvsh\" (UID: \"be2806bc-640c-4904-8f88-8ccc2d58c6ae\") " pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:11:57.851329 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.851282    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be2806bc-640c-4904-8f88-8ccc2d58c6ae-metrics-certs\") pod \"jobset-controller-manager-6fb6bc9cbf-rbvsh\" (UID: \"be2806bc-640c-4904-8f88-8ccc2d58c6ae\") " pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:11:57.851329 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.851319    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/be2806bc-640c-4904-8f88-8ccc2d58c6ae-manager-config\") pod \"jobset-controller-manager-6fb6bc9cbf-rbvsh\" (UID: \"be2806bc-640c-4904-8f88-8ccc2d58c6ae\") " pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:11:57.851982 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.851962    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/be2806bc-640c-4904-8f88-8ccc2d58c6ae-manager-config\") pod \"jobset-controller-manager-6fb6bc9cbf-rbvsh\" (UID: \"be2806bc-640c-4904-8f88-8ccc2d58c6ae\") " pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:11:57.853892 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.853860    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be2806bc-640c-4904-8f88-8ccc2d58c6ae-metrics-certs\") pod \"jobset-controller-manager-6fb6bc9cbf-rbvsh\" (UID: \"be2806bc-640c-4904-8f88-8ccc2d58c6ae\") " pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:11:57.853973 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.853921    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be2806bc-640c-4904-8f88-8ccc2d58c6ae-cert\") pod \"jobset-controller-manager-6fb6bc9cbf-rbvsh\" (UID: \"be2806bc-640c-4904-8f88-8ccc2d58c6ae\") " pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:11:57.865375 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.865351    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zjr2\" (UniqueName: \"kubernetes.io/projected/be2806bc-640c-4904-8f88-8ccc2d58c6ae-kube-api-access-6zjr2\") pod \"jobset-controller-manager-6fb6bc9cbf-rbvsh\" (UID: \"be2806bc-640c-4904-8f88-8ccc2d58c6ae\") " pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:11:57.931949 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:57.931917    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:11:58.176725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:58.176637    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq" event={"ID":"cbeb1f8a-f73e-4b82-bdc5-048c3e57453e","Type":"ContainerStarted","Data":"b136c494b5dddac8cbea8acfe8b1c7609540706d319cc17f7827156e8433637f"}
Nov 03 16:11:58.215143 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:58.215086    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-864dbfdb86-4kdzq" podStartSLOduration=1.688661154 podStartE2EDuration="4.215073312s" totalImagesPullingTime="2.526412158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:58.213708176 +0000 UTC m=+532.227106323" watchObservedRunningTime="2026-11-03 16:11:58.215073312 +0000 UTC m=+532.228471457"
Nov 03 16:11:58.229489 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:58.229457    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"]
Nov 03 16:11:58.234248 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:11:58.234219    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2806bc_640c_4904_8f88_8ccc2d58c6ae.slice/crio-0b7e2ae3f09e539b41236e58f2923a001774c9c6241655679b5cb7879accefbb WatchSource:0}: Error finding container 0b7e2ae3f09e539b41236e58f2923a001774c9c6241655679b5cb7879accefbb: Status 404 returned error can't find the container with id 0b7e2ae3f09e539b41236e58f2923a001774c9c6241655679b5cb7879accefbb
Nov 03 16:11:58.798009 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:11:58.797975    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:11:59.185553 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:11:59.185474    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh" event={"ID":"be2806bc-640c-4904-8f88-8ccc2d58c6ae","Type":"ContainerStarted","Data":"0b7e2ae3f09e539b41236e58f2923a001774c9c6241655679b5cb7879accefbb"}
Nov 03 16:12:01.940401 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:01.940358    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:02.102300 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:02.102252    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:02.203019 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:02.202927    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh" event={"ID":"be2806bc-640c-4904-8f88-8ccc2d58c6ae","Type":"ContainerStarted","Data":"fc5cd7ab52e47745086039ef925a6d923caec782567abff169c4c1c3544aba3a"}
Nov 03 16:12:02.203164 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:02.203020    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:12:02.225352 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:02.225270    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh" podStartSLOduration=2.113490386 podStartE2EDuration="5.225254671s" totalImagesPullingTime="3.111764285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:12:02.224034436 +0000 UTC m=+536.237432582" watchObservedRunningTime="2026-11-03 16:12:02.225254671 +0000 UTC m=+536.238652816"
Nov 03 16:12:04.773908 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:12:04.773866    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8136e22_5d4e_460b_b15e_defa2ca183e4.slice/crio-conmon-03bb422b17bd13387297a508874789432797b1f8e69e721a5719c12d6148a087.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:12:08.612678 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:08.612643    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:08.785347 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:08.785315    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:08.938823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:08.938713    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:11.415095 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:11.415066    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:12.869180 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:12.869148    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:13.213828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:13.213702    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6fb6bc9cbf-rbvsh"
Nov 03 16:12:13.835499 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:13.835469    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:16.240375 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:16.240294    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:16.415939 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:16.415902    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:16.596732 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:16.596702    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:21.452231 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:21.452196    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:21.613943 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:21.613906    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:23.466320 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:23.466286    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:23.622891 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:23.622858    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:28.794805 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:12:28.794754    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:12:33.244715 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:33.244680    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:33.398473 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:33.398410    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:50.956662 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:50.956627    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:51.146424 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:51.146391    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:51.303011 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:51.302977    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:51.455105 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:51.455075    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:52.317155 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:52.317124    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:52.573448 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:52.573364    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:52.741340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:52.741306    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:57.119968 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:57.119924    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:57.278031 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:57.277999    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:57.924999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:57.924964    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lx2gl"]
Nov 03 16:12:57.927621 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:57.927598    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lx2gl"
Nov 03 16:12:57.934201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:57.934073    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 16:12:57.935907 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:57.935885    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lx2gl"]
Nov 03 16:12:57.937507 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:57.937486    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 16:12:57.938940 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:57.938922    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jnwbp\""
Nov 03 16:12:58.046301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:58.046268    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:58.075621 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:58.075592    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdcjm\" (UniqueName: \"kubernetes.io/projected/e08c842e-107a-4c65-9572-960d76451336-kube-api-access-tdcjm\") pod \"authorino-57f7f8659f-lx2gl\" (UID: \"e08c842e-107a-4c65-9572-960d76451336\") " pod="kuadrant-system/authorino-57f7f8659f-lx2gl"
Nov 03 16:12:58.082413 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:58.082386    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8nv7k"]
Nov 03 16:12:58.084825 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:58.084787    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8nv7k"
Nov 03 16:12:58.093698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:58.093677    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8nv7k"]
Nov 03 16:12:58.176489 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:58.176420    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdcjm\" (UniqueName: \"kubernetes.io/projected/e08c842e-107a-4c65-9572-960d76451336-kube-api-access-tdcjm\") pod \"authorino-57f7f8659f-lx2gl\" (UID: \"e08c842e-107a-4c65-9572-960d76451336\") " pod="kuadrant-system/authorino-57f7f8659f-lx2gl"
Nov 03 16:12:58.189201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:58.189172    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdcjm\" (UniqueName: \"kubernetes.io/projected/e08c842e-107a-4c65-9572-960d76451336-kube-api-access-tdcjm\") pod \"authorino-57f7f8659f-lx2gl\" (UID: \"e08c842e-107a-4c65-9572-960d76451336\") " pod="kuadrant-system/authorino-57f7f8659f-lx2gl"
Nov 03 16:12:58.192468 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:58.192447    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:12:58.239711 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:58.239690    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lx2gl"
Nov 03 16:12:58.277261 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:58.277223    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf6cg\" (UniqueName: \"kubernetes.io/projected/0a7c41b3-7b8e-4e5a-893d-f5cdccdad941-kube-api-access-sf6cg\") pod \"authorino-56c6c9ddd5-8nv7k\" (UID: \"0a7c41b3-7b8e-4e5a-893d-f5cdccdad941\") " pod="kuadrant-system/authorino-56c6c9ddd5-8nv7k"
Nov 03 16:12:58.378059 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:58.378026    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf6cg\" (UniqueName: \"kubernetes.io/projected/0a7c41b3-7b8e-4e5a-893d-f5cdccdad941-kube-api-access-sf6cg\") pod \"authorino-56c6c9ddd5-8nv7k\" (UID: \"0a7c41b3-7b8e-4e5a-893d-f5cdccdad941\") " pod="kuadrant-system/authorino-56c6c9ddd5-8nv7k"
Nov 03 16:12:58.390999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:58.390971    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf6cg\" (UniqueName: \"kubernetes.io/projected/0a7c41b3-7b8e-4e5a-893d-f5cdccdad941-kube-api-access-sf6cg\") pod \"authorino-56c6c9ddd5-8nv7k\" (UID: \"0a7c41b3-7b8e-4e5a-893d-f5cdccdad941\") " pod="kuadrant-system/authorino-56c6c9ddd5-8nv7k"
Nov 03 16:12:58.395818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:58.395790    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8nv7k"
Nov 03 16:12:58.510735 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:58.510704    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lx2gl"]
Nov 03 16:12:58.514726 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:12:58.514697    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode08c842e_107a_4c65_9572_960d76451336.slice/crio-46df364564bbfe0cfb1d36b266f0cf5e64f10a472cc9a070ca257e3ec140f0e0 WatchSource:0}: Error finding container 46df364564bbfe0cfb1d36b266f0cf5e64f10a472cc9a070ca257e3ec140f0e0: Status 404 returned error can't find the container with id 46df364564bbfe0cfb1d36b266f0cf5e64f10a472cc9a070ca257e3ec140f0e0
Nov 03 16:12:58.668562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:58.668536    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8nv7k"]
Nov 03 16:12:58.672369 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:12:58.672337    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a7c41b3_7b8e_4e5a_893d_f5cdccdad941.slice/crio-f631287a3688efbfc8f6e304fb1be95f7225e1b685d91388d419a87f552b8a8a WatchSource:0}: Error finding container f631287a3688efbfc8f6e304fb1be95f7225e1b685d91388d419a87f552b8a8a: Status 404 returned error can't find the container with id f631287a3688efbfc8f6e304fb1be95f7225e1b685d91388d419a87f552b8a8a
Nov 03 16:12:58.796981 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:12:58.796944    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:12:59.460115 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:59.460080    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8nv7k" event={"ID":"0a7c41b3-7b8e-4e5a-893d-f5cdccdad941","Type":"ContainerStarted","Data":"f631287a3688efbfc8f6e304fb1be95f7225e1b685d91388d419a87f552b8a8a"}
Nov 03 16:12:59.461208 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:12:59.461180    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lx2gl" event={"ID":"e08c842e-107a-4c65-9572-960d76451336","Type":"ContainerStarted","Data":"46df364564bbfe0cfb1d36b266f0cf5e64f10a472cc9a070ca257e3ec140f0e0"}
Nov 03 16:13:02.479353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:02.479259    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lx2gl" event={"ID":"e08c842e-107a-4c65-9572-960d76451336","Type":"ContainerStarted","Data":"7d1b4e664d84162c8221d313cb674851b42548cfd4e2c7111ed861ecb4568228"}
Nov 03 16:13:02.480779 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:02.480749    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8nv7k" event={"ID":"0a7c41b3-7b8e-4e5a-893d-f5cdccdad941","Type":"ContainerStarted","Data":"d114800a4ea210a7d32920aa506803e88b5821b9d39baa1dd6636d3d051f35ac"}
Nov 03 16:13:02.497909 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:02.497857    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-lx2gl" podStartSLOduration=2.187934458 podStartE2EDuration="5.497842602s" totalImagesPullingTime="3.309908144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:02.497088216 +0000 UTC m=+596.510486345" watchObservedRunningTime="2026-11-03 16:13:02.497842602 +0000 UTC m=+596.511240789"
Nov 03 16:13:02.518217 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:02.518157    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-8nv7k" podStartSLOduration=0.981507226 podStartE2EDuration="4.518136255s" totalImagesPullingTime="3.536629029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:02.516462358 +0000 UTC m=+596.529860505" watchObservedRunningTime="2026-11-03 16:13:02.518136255 +0000 UTC m=+596.531534402"
Nov 03 16:13:02.540609 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:02.540577    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lx2gl"]
Nov 03 16:13:04.489386 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:04.489328    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-lx2gl" podUID="e08c842e-107a-4c65-9572-960d76451336" containerName="authorino" containerID="cri-o://7d1b4e664d84162c8221d313cb674851b42548cfd4e2c7111ed861ecb4568228" gracePeriod=30
Nov 03 16:13:04.731245 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:04.731220    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lx2gl"
Nov 03 16:13:04.830631 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:04.830596    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08c842e-107a-4c65-9572-960d76451336-kube-api-access-tdcjm\" (UniqueName: \"kubernetes.io/projected/e08c842e-107a-4c65-9572-960d76451336-kube-api-access-tdcjm\") pod \"e08c842e-107a-4c65-9572-960d76451336\" (UID: \"e08c842e-107a-4c65-9572-960d76451336\") "
Nov 03 16:13:04.832895 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:04.832871    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08c842e-107a-4c65-9572-960d76451336-kube-api-access-tdcjm" pod "e08c842e-107a-4c65-9572-960d76451336" (UID: "e08c842e-107a-4c65-9572-960d76451336"). InnerVolumeSpecName "kube-api-access-tdcjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:13:04.932152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:04.932117    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdcjm\" (UniqueName: \"kubernetes.io/projected/e08c842e-107a-4c65-9572-960d76451336-kube-api-access-tdcjm\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:13:05.498087 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:05.498047    6506 generic.go:356] "Generic (PLEG): container finished" podID="e08c842e-107a-4c65-9572-960d76451336" containerID="7d1b4e664d84162c8221d313cb674851b42548cfd4e2c7111ed861ecb4568228" exitCode=0
Nov 03 16:13:05.498500 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:05.498110    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lx2gl"
Nov 03 16:13:05.498500 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:05.498137    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lx2gl" event={"ID":"e08c842e-107a-4c65-9572-960d76451336","Type":"ContainerDied","Data":"7d1b4e664d84162c8221d313cb674851b42548cfd4e2c7111ed861ecb4568228"}
Nov 03 16:13:05.498500 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:05.498191    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lx2gl" event={"ID":"e08c842e-107a-4c65-9572-960d76451336","Type":"ContainerDied","Data":"46df364564bbfe0cfb1d36b266f0cf5e64f10a472cc9a070ca257e3ec140f0e0"}
Nov 03 16:13:05.498500 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:05.498212    6506 scope.go:122] "RemoveContainer" containerID="7d1b4e664d84162c8221d313cb674851b42548cfd4e2c7111ed861ecb4568228"
Nov 03 16:13:05.506712 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:05.506694    6506 scope.go:122] "RemoveContainer" containerID="7d1b4e664d84162c8221d313cb674851b42548cfd4e2c7111ed861ecb4568228"
Nov 03 16:13:05.506990 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:13:05.506972    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d1b4e664d84162c8221d313cb674851b42548cfd4e2c7111ed861ecb4568228\": container with ID starting with 7d1b4e664d84162c8221d313cb674851b42548cfd4e2c7111ed861ecb4568228 not found: ID does not exist" containerID="7d1b4e664d84162c8221d313cb674851b42548cfd4e2c7111ed861ecb4568228"
Nov 03 16:13:05.507056 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:05.506998    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d1b4e664d84162c8221d313cb674851b42548cfd4e2c7111ed861ecb4568228"} err="failed to get container status \"7d1b4e664d84162c8221d313cb674851b42548cfd4e2c7111ed861ecb4568228\": rpc error: code = NotFound desc = could not find container \"7d1b4e664d84162c8221d313cb674851b42548cfd4e2c7111ed861ecb4568228\": container with ID starting with 7d1b4e664d84162c8221d313cb674851b42548cfd4e2c7111ed861ecb4568228 not found: ID does not exist"
Nov 03 16:13:05.519295 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:05.519271    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lx2gl"]
Nov 03 16:13:05.523408 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:05.523390    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lx2gl"]
Nov 03 16:13:06.101609 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:06.101573    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e08c842e-107a-4c65-9572-960d76451336" path="/var/lib/kubelet/pods/e08c842e-107a-4c65-9572-960d76451336/volumes"
Nov 03 16:13:08.896951 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:08.896915    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:13:13.845298 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:13.845263    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:13:14.037403 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:14.037362    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:13:14.695184 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:14.695150    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:13:24.122711 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:24.122679    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:13:24.270434 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:24.270404    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:13:28.795459 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:13:28.795422    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:13:34.226870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:34.226836    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:13:34.409689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:34.409656    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:13:34.591268 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:34.591240    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:13:38.088537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:38.088505    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:13:38.255714 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:38.255682    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:13:39.762908 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:39.762869    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:13:39.926342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:13:39.926309    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:13:58.796073 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:13:58.795993    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:14:08.181889 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:08.181850    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:14:08.334773 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:08.334737    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:14:08.940387 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:08.940358    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:14:09.393784 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:09.393751    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:14:09.543639 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:09.543606    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:14:09.757000 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:09.756927    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:14:17.745961 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:17.745923    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:14:17.746373 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:17.746007    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz" podUID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:14:19.657033 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:19.656998    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:14:19.855546 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:19.855515    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:14:19.856713 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:19.856687    6506 generic.go:356] "Generic (PLEG): container finished" podID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" containerID="58e19e8ed66f23c5896cde17db008e9264921b1577f320b0d3ca6a554744c5e9" exitCode=0
Nov 03 16:14:19.856874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:19.856760    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz" event={"ID":"52baf237-4a39-4822-b1b4-4d9a6bd36b85","Type":"ContainerDied","Data":"58e19e8ed66f23c5896cde17db008e9264921b1577f320b0d3ca6a554744c5e9"}
Nov 03 16:14:19.857254 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:19.857119    6506 scope.go:122] "RemoveContainer" containerID="58e19e8ed66f23c5896cde17db008e9264921b1577f320b0d3ca6a554744c5e9"
Nov 03 16:14:19.858197 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:19.858170    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:14:19.858274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:19.858261    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:14:19.858314 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:19.858296    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:14:19.905294 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:19.905274    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tlkh9"]
Nov 03 16:14:19.905629 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:19.905608    6506 cpu_manager.go:414] "containerMap: removing container" podUID="e08c842e-107a-4c65-9572-960d76451336" containerName="authorino"
Nov 03 16:14:19.905629 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:19.905628    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08c842e-107a-4c65-9572-960d76451336" containerName="authorino"
Nov 03 16:14:19.905760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:19.905716    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08c842e-107a-4c65-9572-960d76451336" containerName="authorino"
Nov 03 16:14:19.908289 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:19.908236    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tlkh9"
Nov 03 16:14:19.919662 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:19.919083    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tlkh9"]
Nov 03 16:14:19.945301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:19.945260    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd8q7\" (UniqueName: \"kubernetes.io/projected/a12b58e8-11de-4baa-a061-127d29132e0b-kube-api-access-gd8q7\") pod \"authorino-6f6944bcdd-tlkh9\" (UID: \"a12b58e8-11de-4baa-a061-127d29132e0b\") " pod="kuadrant-system/authorino-6f6944bcdd-tlkh9"
Nov 03 16:14:20.008774 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.008742    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tlkh9"]
Nov 03 16:14:20.009085 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:14:20.009059    6506 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gd8q7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-tlkh9" podUID="a12b58e8-11de-4baa-a061-127d29132e0b"
Nov 03 16:14:20.041138 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.041111    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-78swk"]
Nov 03 16:14:20.043458 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.043436    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-78swk"
Nov 03 16:14:20.046229 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.046199    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd8q7\" (UniqueName: \"kubernetes.io/projected/a12b58e8-11de-4baa-a061-127d29132e0b-kube-api-access-gd8q7\") pod \"authorino-6f6944bcdd-tlkh9\" (UID: \"a12b58e8-11de-4baa-a061-127d29132e0b\") " pod="kuadrant-system/authorino-6f6944bcdd-tlkh9"
Nov 03 16:14:20.050266 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.050247    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 16:14:20.058929 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.058896    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd8q7\" (UniqueName: \"kubernetes.io/projected/a12b58e8-11de-4baa-a061-127d29132e0b-kube-api-access-gd8q7\") pod \"authorino-6f6944bcdd-tlkh9\" (UID: \"a12b58e8-11de-4baa-a061-127d29132e0b\") " pod="kuadrant-system/authorino-6f6944bcdd-tlkh9"
Nov 03 16:14:20.062427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.062407    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-78swk"]
Nov 03 16:14:20.147605 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.147576    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45mnh\" (UniqueName: \"kubernetes.io/projected/5d7565a2-c58e-45e8-b997-de97f14bf4cf-kube-api-access-45mnh\") pod \"authorino-6889cbd8c4-78swk\" (UID: \"5d7565a2-c58e-45e8-b997-de97f14bf4cf\") " pod="kuadrant-system/authorino-6889cbd8c4-78swk"
Nov 03 16:14:20.147768 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.147628    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d7565a2-c58e-45e8-b997-de97f14bf4cf-tls-cert\") pod \"authorino-6889cbd8c4-78swk\" (UID: \"5d7565a2-c58e-45e8-b997-de97f14bf4cf\") " pod="kuadrant-system/authorino-6889cbd8c4-78swk"
Nov 03 16:14:20.248292 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.248210    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45mnh\" (UniqueName: \"kubernetes.io/projected/5d7565a2-c58e-45e8-b997-de97f14bf4cf-kube-api-access-45mnh\") pod \"authorino-6889cbd8c4-78swk\" (UID: \"5d7565a2-c58e-45e8-b997-de97f14bf4cf\") " pod="kuadrant-system/authorino-6889cbd8c4-78swk"
Nov 03 16:14:20.248292 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.248266    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d7565a2-c58e-45e8-b997-de97f14bf4cf-tls-cert\") pod \"authorino-6889cbd8c4-78swk\" (UID: \"5d7565a2-c58e-45e8-b997-de97f14bf4cf\") " pod="kuadrant-system/authorino-6889cbd8c4-78swk"
Nov 03 16:14:20.251016 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.250985    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d7565a2-c58e-45e8-b997-de97f14bf4cf-tls-cert\") pod \"authorino-6889cbd8c4-78swk\" (UID: \"5d7565a2-c58e-45e8-b997-de97f14bf4cf\") " pod="kuadrant-system/authorino-6889cbd8c4-78swk"
Nov 03 16:14:20.267072 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.267040    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45mnh\" (UniqueName: \"kubernetes.io/projected/5d7565a2-c58e-45e8-b997-de97f14bf4cf-kube-api-access-45mnh\") pod \"authorino-6889cbd8c4-78swk\" (UID: \"5d7565a2-c58e-45e8-b997-de97f14bf4cf\") " pod="kuadrant-system/authorino-6889cbd8c4-78swk"
Nov 03 16:14:20.279164 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.279139    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-78swk"]
Nov 03 16:14:20.279340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.279329    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-78swk"
Nov 03 16:14:20.327824 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.313379    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66467df5bb-hg9qx"]
Nov 03 16:14:20.336773 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.336600    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66467df5bb-hg9qx"]
Nov 03 16:14:20.336773 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.336772    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66467df5bb-hg9qx"
Nov 03 16:14:20.450003 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.449972    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n64ts\" (UniqueName: \"kubernetes.io/projected/f1812e02-c0db-4ea6-9382-e0331ba89300-kube-api-access-n64ts\") pod \"authorino-66467df5bb-hg9qx\" (UID: \"f1812e02-c0db-4ea6-9382-e0331ba89300\") " pod="kuadrant-system/authorino-66467df5bb-hg9qx"
Nov 03 16:14:20.450181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.450023    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1812e02-c0db-4ea6-9382-e0331ba89300-tls-cert\") pod \"authorino-66467df5bb-hg9qx\" (UID: \"f1812e02-c0db-4ea6-9382-e0331ba89300\") " pod="kuadrant-system/authorino-66467df5bb-hg9qx"
Nov 03 16:14:20.550515 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.550481    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1812e02-c0db-4ea6-9382-e0331ba89300-tls-cert\") pod \"authorino-66467df5bb-hg9qx\" (UID: \"f1812e02-c0db-4ea6-9382-e0331ba89300\") " pod="kuadrant-system/authorino-66467df5bb-hg9qx"
Nov 03 16:14:20.550700 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.550552    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n64ts\" (UniqueName: \"kubernetes.io/projected/f1812e02-c0db-4ea6-9382-e0331ba89300-kube-api-access-n64ts\") pod \"authorino-66467df5bb-hg9qx\" (UID: \"f1812e02-c0db-4ea6-9382-e0331ba89300\") " pod="kuadrant-system/authorino-66467df5bb-hg9qx"
Nov 03 16:14:20.553069 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.553036    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1812e02-c0db-4ea6-9382-e0331ba89300-tls-cert\") pod \"authorino-66467df5bb-hg9qx\" (UID: \"f1812e02-c0db-4ea6-9382-e0331ba89300\") " pod="kuadrant-system/authorino-66467df5bb-hg9qx"
Nov 03 16:14:20.564091 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.564066    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n64ts\" (UniqueName: \"kubernetes.io/projected/f1812e02-c0db-4ea6-9382-e0331ba89300-kube-api-access-n64ts\") pod \"authorino-66467df5bb-hg9qx\" (UID: \"f1812e02-c0db-4ea6-9382-e0331ba89300\") " pod="kuadrant-system/authorino-66467df5bb-hg9qx"
Nov 03 16:14:20.581166 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.581147    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-78swk"]
Nov 03 16:14:20.583152 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:14:20.583127    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d7565a2_c58e_45e8_b997_de97f14bf4cf.slice/crio-137d6687cb8a42404ab15640c768647ad4d8d760caaad47cba4d24025d966efe WatchSource:0}: Error finding container 137d6687cb8a42404ab15640c768647ad4d8d760caaad47cba4d24025d966efe: Status 404 returned error can't find the container with id 137d6687cb8a42404ab15640c768647ad4d8d760caaad47cba4d24025d966efe
Nov 03 16:14:20.650945 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.650916    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66467df5bb-hg9qx"
Nov 03 16:14:20.863381 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.863282    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz" event={"ID":"52baf237-4a39-4822-b1b4-4d9a6bd36b85","Type":"ContainerStarted","Data":"534bbf6cd27b3d9c1333058425948734bf1a2a66efcf70b283b20fe2b5df370f"}
Nov 03 16:14:20.864543 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.864524    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tlkh9"
Nov 03 16:14:20.864543 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.864530    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-78swk" event={"ID":"5d7565a2-c58e-45e8-b997-de97f14bf4cf","Type":"ContainerStarted","Data":"137d6687cb8a42404ab15640c768647ad4d8d760caaad47cba4d24025d966efe"}
Nov 03 16:14:20.869936 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.869919    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tlkh9"
Nov 03 16:14:20.934937 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.934914    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66467df5bb-hg9qx"]
Nov 03 16:14:20.936743 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:14:20.936714    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1812e02_c0db_4ea6_9382_e0331ba89300.slice/crio-6d1e5f979a8d8f09de370911e36137aec233967f3334a617f2fad8ee29b99b23 WatchSource:0}: Error finding container 6d1e5f979a8d8f09de370911e36137aec233967f3334a617f2fad8ee29b99b23: Status 404 returned error can't find the container with id 6d1e5f979a8d8f09de370911e36137aec233967f3334a617f2fad8ee29b99b23
Nov 03 16:14:20.954634 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.954606    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a12b58e8-11de-4baa-a061-127d29132e0b-kube-api-access-gd8q7\" (UniqueName: \"kubernetes.io/projected/a12b58e8-11de-4baa-a061-127d29132e0b-kube-api-access-gd8q7\") pod \"a12b58e8-11de-4baa-a061-127d29132e0b\" (UID: \"a12b58e8-11de-4baa-a061-127d29132e0b\") "
Nov 03 16:14:20.956886 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:20.956858    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12b58e8-11de-4baa-a061-127d29132e0b-kube-api-access-gd8q7" pod "a12b58e8-11de-4baa-a061-127d29132e0b" (UID: "a12b58e8-11de-4baa-a061-127d29132e0b"). InnerVolumeSpecName "kube-api-access-gd8q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:14:21.055977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:21.055943    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd8q7\" (UniqueName: \"kubernetes.io/projected/a12b58e8-11de-4baa-a061-127d29132e0b-kube-api-access-gd8q7\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:14:21.850394 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:21.850370    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:14:21.850669 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:21.850643    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Nov 03 16:14:21.850755 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:21.850706    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz" podUID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Nov 03 16:14:21.871465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:21.871438    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-78swk" event={"ID":"5d7565a2-c58e-45e8-b997-de97f14bf4cf","Type":"ContainerStarted","Data":"b7d38cf65a6b57d1a5c326b9d014e596cc78038e6cd9adad621e80a511bada09"}
Nov 03 16:14:21.871824 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:21.871540    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-78swk" podUID="5d7565a2-c58e-45e8-b997-de97f14bf4cf" containerName="authorino" containerID="cri-o://b7d38cf65a6b57d1a5c326b9d014e596cc78038e6cd9adad621e80a511bada09" gracePeriod=30
Nov 03 16:14:21.873370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:21.873343    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66467df5bb-hg9qx" event={"ID":"f1812e02-c0db-4ea6-9382-e0331ba89300","Type":"ContainerStarted","Data":"d7b809b5264d308717a5ba88b6902b03f521f701abb97b069568d5e5b1836bff"}
Nov 03 16:14:21.873475 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:21.873384    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66467df5bb-hg9qx" event={"ID":"f1812e02-c0db-4ea6-9382-e0331ba89300","Type":"ContainerStarted","Data":"6d1e5f979a8d8f09de370911e36137aec233967f3334a617f2fad8ee29b99b23"}
Nov 03 16:14:21.873475 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:21.873413    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tlkh9"
Nov 03 16:14:21.890784 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:21.890730    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-78swk" podStartSLOduration=0.707305599 podStartE2EDuration="1.890711407s" totalImagesPullingTime="1.183405808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:21.889989804 +0000 UTC m=+675.903387951" watchObservedRunningTime="2026-11-03 16:14:21.890711407 +0000 UTC m=+675.904109555"
Nov 03 16:14:21.940623 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:21.940586    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tlkh9"]
Nov 03 16:14:21.940780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:21.940638    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tlkh9"]
Nov 03 16:14:21.961488 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:21.961409    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-66467df5bb-hg9qx" podStartSLOduration=1.134637538 podStartE2EDuration="1.961387077s" totalImagesPullingTime="826.749539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:21.959384071 +0000 UTC m=+675.972782215" watchObservedRunningTime="2026-11-03 16:14:21.961387077 +0000 UTC m=+675.974785230"
Nov 03 16:14:21.994181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:21.991156    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8nv7k"]
Nov 03 16:14:21.994181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:21.991327    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-8nv7k" podUID="0a7c41b3-7b8e-4e5a-893d-f5cdccdad941" containerName="authorino" containerID="cri-o://d114800a4ea210a7d32920aa506803e88b5821b9d39baa1dd6636d3d051f35ac" gracePeriod=30
Nov 03 16:14:22.131836 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.106926    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a12b58e8-11de-4baa-a061-127d29132e0b" path="/var/lib/kubelet/pods/a12b58e8-11de-4baa-a061-127d29132e0b/volumes"
Nov 03 16:14:22.300929 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.300901    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8nv7k"
Nov 03 16:14:22.371063 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.368701    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7c41b3-7b8e-4e5a-893d-f5cdccdad941-kube-api-access-sf6cg\" (UniqueName: \"kubernetes.io/projected/0a7c41b3-7b8e-4e5a-893d-f5cdccdad941-kube-api-access-sf6cg\") pod \"0a7c41b3-7b8e-4e5a-893d-f5cdccdad941\" (UID: \"0a7c41b3-7b8e-4e5a-893d-f5cdccdad941\") "
Nov 03 16:14:22.374254 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.374215    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7c41b3-7b8e-4e5a-893d-f5cdccdad941-kube-api-access-sf6cg" pod "0a7c41b3-7b8e-4e5a-893d-f5cdccdad941" (UID: "0a7c41b3-7b8e-4e5a-893d-f5cdccdad941"). InnerVolumeSpecName "kube-api-access-sf6cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:14:22.469977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.469936    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf6cg\" (UniqueName: \"kubernetes.io/projected/0a7c41b3-7b8e-4e5a-893d-f5cdccdad941-kube-api-access-sf6cg\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:14:22.850484 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.850441    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Nov 03 16:14:22.850659 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.850514    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz" podUID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Nov 03 16:14:22.883862 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.883825    6506 generic.go:356] "Generic (PLEG): container finished" podID="0a7c41b3-7b8e-4e5a-893d-f5cdccdad941" containerID="d114800a4ea210a7d32920aa506803e88b5821b9d39baa1dd6636d3d051f35ac" exitCode=0
Nov 03 16:14:22.884974 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.884386    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8nv7k"
Nov 03 16:14:22.889132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.884401    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8nv7k" event={"ID":"0a7c41b3-7b8e-4e5a-893d-f5cdccdad941","Type":"ContainerDied","Data":"d114800a4ea210a7d32920aa506803e88b5821b9d39baa1dd6636d3d051f35ac"}
Nov 03 16:14:22.889132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.885907    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8nv7k" event={"ID":"0a7c41b3-7b8e-4e5a-893d-f5cdccdad941","Type":"ContainerDied","Data":"f631287a3688efbfc8f6e304fb1be95f7225e1b685d91388d419a87f552b8a8a"}
Nov 03 16:14:22.889132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.885937    6506 scope.go:122] "RemoveContainer" containerID="d114800a4ea210a7d32920aa506803e88b5821b9d39baa1dd6636d3d051f35ac"
Nov 03 16:14:22.899855 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.899833    6506 scope.go:122] "RemoveContainer" containerID="d114800a4ea210a7d32920aa506803e88b5821b9d39baa1dd6636d3d051f35ac"
Nov 03 16:14:22.900173 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:14:22.900152    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d114800a4ea210a7d32920aa506803e88b5821b9d39baa1dd6636d3d051f35ac\": container with ID starting with d114800a4ea210a7d32920aa506803e88b5821b9d39baa1dd6636d3d051f35ac not found: ID does not exist" containerID="d114800a4ea210a7d32920aa506803e88b5821b9d39baa1dd6636d3d051f35ac"
Nov 03 16:14:22.900244 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.900185    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d114800a4ea210a7d32920aa506803e88b5821b9d39baa1dd6636d3d051f35ac"} err="failed to get container status \"d114800a4ea210a7d32920aa506803e88b5821b9d39baa1dd6636d3d051f35ac\": rpc error: code = NotFound desc = could not find container \"d114800a4ea210a7d32920aa506803e88b5821b9d39baa1dd6636d3d051f35ac\": container with ID starting with d114800a4ea210a7d32920aa506803e88b5821b9d39baa1dd6636d3d051f35ac not found: ID does not exist"
Nov 03 16:14:22.933881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.913377    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8nv7k"]
Nov 03 16:14:22.933881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:22.933229    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8nv7k"]
Nov 03 16:14:23.850585 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:23.850542    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Nov 03 16:14:23.850735 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:23.850608    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz" podUID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Nov 03 16:14:24.102109 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:24.102036    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a7c41b3-7b8e-4e5a-893d-f5cdccdad941" path="/var/lib/kubelet/pods/0a7c41b3-7b8e-4e5a-893d-f5cdccdad941/volumes"
Nov 03 16:14:24.849774 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:24.849740    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Nov 03 16:14:24.849963 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:24.849821    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz" podUID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Nov 03 16:14:25.896392 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:25.896361    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:14:25.896897 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:25.896761    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:14:25.897772 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:25.897752    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:14:28.796246 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:14:28.796209    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:14:52.019024 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:52.018992    6506 generic.go:356] "Generic (PLEG): container finished" podID="5d7565a2-c58e-45e8-b997-de97f14bf4cf" containerID="b7d38cf65a6b57d1a5c326b9d014e596cc78038e6cd9adad621e80a511bada09" exitCode=137
Nov 03 16:14:52.019418 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:52.019045    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-78swk" event={"ID":"5d7565a2-c58e-45e8-b997-de97f14bf4cf","Type":"ContainerDied","Data":"b7d38cf65a6b57d1a5c326b9d014e596cc78038e6cd9adad621e80a511bada09"}
Nov 03 16:14:52.512080 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:52.512058    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-78swk"
Nov 03 16:14:52.631645 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:52.631609    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7565a2-c58e-45e8-b997-de97f14bf4cf-kube-api-access-45mnh\" (UniqueName: \"kubernetes.io/projected/5d7565a2-c58e-45e8-b997-de97f14bf4cf-kube-api-access-45mnh\") pod \"5d7565a2-c58e-45e8-b997-de97f14bf4cf\" (UID: \"5d7565a2-c58e-45e8-b997-de97f14bf4cf\") "
Nov 03 16:14:52.631645 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:52.631647    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d7565a2-c58e-45e8-b997-de97f14bf4cf-tls-cert\" (UniqueName: \"kubernetes.io/secret/5d7565a2-c58e-45e8-b997-de97f14bf4cf-tls-cert\") pod \"5d7565a2-c58e-45e8-b997-de97f14bf4cf\" (UID: \"5d7565a2-c58e-45e8-b997-de97f14bf4cf\") "
Nov 03 16:14:52.633902 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:52.633879    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7565a2-c58e-45e8-b997-de97f14bf4cf-kube-api-access-45mnh" pod "5d7565a2-c58e-45e8-b997-de97f14bf4cf" (UID: "5d7565a2-c58e-45e8-b997-de97f14bf4cf"). InnerVolumeSpecName "kube-api-access-45mnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:14:52.641518 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:52.641493    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7565a2-c58e-45e8-b997-de97f14bf4cf-tls-cert" pod "5d7565a2-c58e-45e8-b997-de97f14bf4cf" (UID: "5d7565a2-c58e-45e8-b997-de97f14bf4cf"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:14:52.733155 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:52.733117    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45mnh\" (UniqueName: \"kubernetes.io/projected/5d7565a2-c58e-45e8-b997-de97f14bf4cf-kube-api-access-45mnh\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:14:52.733155 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:52.733148    6506 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d7565a2-c58e-45e8-b997-de97f14bf4cf-tls-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:14:53.024589 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:53.024558    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-78swk" event={"ID":"5d7565a2-c58e-45e8-b997-de97f14bf4cf","Type":"ContainerDied","Data":"137d6687cb8a42404ab15640c768647ad4d8d760caaad47cba4d24025d966efe"}
Nov 03 16:14:53.025033 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:53.024593    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-78swk"
Nov 03 16:14:53.025033 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:53.024605    6506 scope.go:122] "RemoveContainer" containerID="b7d38cf65a6b57d1a5c326b9d014e596cc78038e6cd9adad621e80a511bada09"
Nov 03 16:14:53.050125 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:53.050101    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-78swk"]
Nov 03 16:14:53.053855 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:53.053836    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-78swk"]
Nov 03 16:14:54.101678 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:54.101638    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7565a2-c58e-45e8-b997-de97f14bf4cf" path="/var/lib/kubelet/pods/5d7565a2-c58e-45e8-b997-de97f14bf4cf/volumes"
Nov 03 16:14:55.996514 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:55.996480    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:14:56.170736 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:14:56.170710    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:14:58.796390 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:14:58.796355    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:15:00.145528 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.145498    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"]
Nov 03 16:15:00.145929 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.145840    6506 cpu_manager.go:414] "containerMap: removing container" podUID="0a7c41b3-7b8e-4e5a-893d-f5cdccdad941" containerName="authorino"
Nov 03 16:15:00.145929 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.145852    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7c41b3-7b8e-4e5a-893d-f5cdccdad941" containerName="authorino"
Nov 03 16:15:00.145929 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.145864    6506 cpu_manager.go:414] "containerMap: removing container" podUID="5d7565a2-c58e-45e8-b997-de97f14bf4cf" containerName="authorino"
Nov 03 16:15:00.145929 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.145872    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7565a2-c58e-45e8-b997-de97f14bf4cf" containerName="authorino"
Nov 03 16:15:00.146064 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.145956    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7c41b3-7b8e-4e5a-893d-f5cdccdad941" containerName="authorino"
Nov 03 16:15:00.146064 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.145964    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7565a2-c58e-45e8-b997-de97f14bf4cf" containerName="authorino"
Nov 03 16:15:00.150251 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.150231    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"
Nov 03 16:15:00.156174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.156139    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:15:00.156378 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.156266    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:15:00.157993 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.157972    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:15:00.160653 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.160630    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"]
Nov 03 16:15:00.299864 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.299825    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fmgr\" (UniqueName: \"kubernetes.io/projected/73ae3a99-2617-44be-9b8d-89b102f079d0-kube-api-access-2fmgr\") pod \"maas-api-key-cleanup-29895375-rhqkk\" (UID: \"73ae3a99-2617-44be-9b8d-89b102f079d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"
Nov 03 16:15:00.300026 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.299887    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ae3a99-2617-44be-9b8d-89b102f079d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895375-rhqkk\" (UID: \"73ae3a99-2617-44be-9b8d-89b102f079d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"
Nov 03 16:15:00.400995 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.400926    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fmgr\" (UniqueName: \"kubernetes.io/projected/73ae3a99-2617-44be-9b8d-89b102f079d0-kube-api-access-2fmgr\") pod \"maas-api-key-cleanup-29895375-rhqkk\" (UID: \"73ae3a99-2617-44be-9b8d-89b102f079d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"
Nov 03 16:15:00.400995 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.400969    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ae3a99-2617-44be-9b8d-89b102f079d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895375-rhqkk\" (UID: \"73ae3a99-2617-44be-9b8d-89b102f079d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"
Nov 03 16:15:00.401535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.401517    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ae3a99-2617-44be-9b8d-89b102f079d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895375-rhqkk\" (UID: \"73ae3a99-2617-44be-9b8d-89b102f079d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"
Nov 03 16:15:00.414408 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.414385    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fmgr\" (UniqueName: \"kubernetes.io/projected/73ae3a99-2617-44be-9b8d-89b102f079d0-kube-api-access-2fmgr\") pod \"maas-api-key-cleanup-29895375-rhqkk\" (UID: \"73ae3a99-2617-44be-9b8d-89b102f079d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"
Nov 03 16:15:00.470237 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.470208    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"
Nov 03 16:15:00.747240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:00.747218    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"]
Nov 03 16:15:00.749331 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:15:00.749305    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ae3a99_2617_44be_9b8d_89b102f079d0.slice/crio-7a5ec00a9a83cb8565565d418dff033c2feef6231fb6b7b5bbba1719c77d1f96 WatchSource:0}: Error finding container 7a5ec00a9a83cb8565565d418dff033c2feef6231fb6b7b5bbba1719c77d1f96: Status 404 returned error can't find the container with id 7a5ec00a9a83cb8565565d418dff033c2feef6231fb6b7b5bbba1719c77d1f96
Nov 03 16:15:01.065281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:01.065241    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk" event={"ID":"73ae3a99-2617-44be-9b8d-89b102f079d0","Type":"ContainerStarted","Data":"7a5ec00a9a83cb8565565d418dff033c2feef6231fb6b7b5bbba1719c77d1f96"}
Nov 03 16:15:09.018532 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:09.018496    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:15:14.223653 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:14.223622    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:15:14.987951 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:14.987914    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"]
Nov 03 16:15:14.995642 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:14.995071    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.015785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.015754    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 16:15:15.015970 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.015892    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 03 16:15:15.015970 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.015966    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qbd52\""
Nov 03 16:15:15.016099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.016056    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 16:15:15.017785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.017756    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc91de6-75b4-489b-9f16-c7175b27e41f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.018039 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.018019    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc91de6-75b4-489b-9f16-c7175b27e41f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.018998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.018970    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc91de6-75b4-489b-9f16-c7175b27e41f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.019103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.019018    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc91de6-75b4-489b-9f16-c7175b27e41f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.019103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.019086    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjfwv\" (UniqueName: \"kubernetes.io/projected/1bc91de6-75b4-489b-9f16-c7175b27e41f-kube-api-access-fjfwv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.019232 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.019149    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc91de6-75b4-489b-9f16-c7175b27e41f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.019592 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.019571    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"]
Nov 03 16:15:15.105079 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.105035    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:15:15.120370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.120328    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc91de6-75b4-489b-9f16-c7175b27e41f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.120557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.120437    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc91de6-75b4-489b-9f16-c7175b27e41f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.120557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.120473    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc91de6-75b4-489b-9f16-c7175b27e41f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.120557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.120535    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc91de6-75b4-489b-9f16-c7175b27e41f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.120739 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.120587    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc91de6-75b4-489b-9f16-c7175b27e41f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.120739 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.120651    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjfwv\" (UniqueName: \"kubernetes.io/projected/1bc91de6-75b4-489b-9f16-c7175b27e41f-kube-api-access-fjfwv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.121435 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.121407    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc91de6-75b4-489b-9f16-c7175b27e41f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.122389 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.122362    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc91de6-75b4-489b-9f16-c7175b27e41f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.122646 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.122622    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc91de6-75b4-489b-9f16-c7175b27e41f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.124623 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.124601    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc91de6-75b4-489b-9f16-c7175b27e41f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.126745 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.126719    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc91de6-75b4-489b-9f16-c7175b27e41f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.134544 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.134516    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjfwv\" (UniqueName: \"kubernetes.io/projected/1bc91de6-75b4-489b-9f16-c7175b27e41f-kube-api-access-fjfwv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr\" (UID: \"1bc91de6-75b4-489b-9f16-c7175b27e41f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.324716 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.324684    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:15.622328 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.622301    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"]
Nov 03 16:15:15.625268 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:15:15.625230    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc91de6_75b4_489b_9f16_c7175b27e41f.slice/crio-9c399a87911820874c1465fb3af2b85c40784bb0dae00f2d90e33fc5855adb2a WatchSource:0}: Error finding container 9c399a87911820874c1465fb3af2b85c40784bb0dae00f2d90e33fc5855adb2a: Status 404 returned error can't find the container with id 9c399a87911820874c1465fb3af2b85c40784bb0dae00f2d90e33fc5855adb2a
Nov 03 16:15:15.663659 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:15.663627    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:15:16.142563 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:16.142473    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr" event={"ID":"1bc91de6-75b4-489b-9f16-c7175b27e41f","Type":"ContainerStarted","Data":"9c399a87911820874c1465fb3af2b85c40784bb0dae00f2d90e33fc5855adb2a"}
Nov 03 16:15:18.157132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:18.157087    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr" event={"ID":"1bc91de6-75b4-489b-9f16-c7175b27e41f","Type":"ContainerStarted","Data":"bdabd1f218b5b7f22bd9de5fac8395bd6d47cfbb4dc8909446b3556b36d451e1"}
Nov 03 16:15:18.185761 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:18.185713    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr" podStartSLOduration=2.516588494 podStartE2EDuration="4.185698123s" totalImagesPullingTime="1.669109629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:18.184667017 +0000 UTC m=+732.198065163" watchObservedRunningTime="2026-11-03 16:15:18.185698123 +0000 UTC m=+732.199096268"
Nov 03 16:15:18.272787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:18.272755    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:15:18.325856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:18.325823    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:19.165060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.165026    6506 generic.go:356] "Generic (PLEG): container finished" podID="73ae3a99-2617-44be-9b8d-89b102f079d0" containerID="4563a2506dd2ded5e75aa7313dd47972a2672db543032695ba3f65dac4e6f718" exitCode=60
Nov 03 16:15:19.165552 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.165106    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk" event={"ID":"73ae3a99-2617-44be-9b8d-89b102f079d0","Type":"ContainerDied","Data":"4563a2506dd2ded5e75aa7313dd47972a2672db543032695ba3f65dac4e6f718"}
Nov 03 16:15:19.165552 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.165365    6506 scope.go:122] "RemoveContainer" containerID="4563a2506dd2ded5e75aa7313dd47972a2672db543032695ba3f65dac4e6f718"
Nov 03 16:15:19.386830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.385554    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"]
Nov 03 16:15:19.411465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.411439    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"]
Nov 03 16:15:19.411763 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.411718    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.439811 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.436267    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 16:15:19.479236 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.479204    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28df0487-be29-4efe-99d6-94b6166fa1cc-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.479446 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.479238    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28df0487-be29-4efe-99d6-94b6166fa1cc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.479446 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.479396    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28df0487-be29-4efe-99d6-94b6166fa1cc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.479446 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.479442    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg8dq\" (UniqueName: \"kubernetes.io/projected/28df0487-be29-4efe-99d6-94b6166fa1cc-kube-api-access-wg8dq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.479616 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.479533    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28df0487-be29-4efe-99d6-94b6166fa1cc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.479616 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.479566    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28df0487-be29-4efe-99d6-94b6166fa1cc-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.580892 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.580779    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28df0487-be29-4efe-99d6-94b6166fa1cc-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.580892 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.580839    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28df0487-be29-4efe-99d6-94b6166fa1cc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.580892 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.580876    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28df0487-be29-4efe-99d6-94b6166fa1cc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.581185 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.580900    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg8dq\" (UniqueName: \"kubernetes.io/projected/28df0487-be29-4efe-99d6-94b6166fa1cc-kube-api-access-wg8dq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.581185 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.580966    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28df0487-be29-4efe-99d6-94b6166fa1cc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.581185 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.581008    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28df0487-be29-4efe-99d6-94b6166fa1cc-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.581428 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.581370    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28df0487-be29-4efe-99d6-94b6166fa1cc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.581675 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.581655    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28df0487-be29-4efe-99d6-94b6166fa1cc-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.581849 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.581827    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28df0487-be29-4efe-99d6-94b6166fa1cc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.583336 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.583313    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28df0487-be29-4efe-99d6-94b6166fa1cc-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.583917 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.583900    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28df0487-be29-4efe-99d6-94b6166fa1cc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.591292 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.591251    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:15:19.597256 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.597234    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg8dq\" (UniqueName: \"kubernetes.io/projected/28df0487-be29-4efe-99d6-94b6166fa1cc-kube-api-access-wg8dq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2\" (UID: \"28df0487-be29-4efe-99d6-94b6166fa1cc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:19.736342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:19.736304    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:20.027186 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:20.024321    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"]
Nov 03 16:15:20.027737 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:15:20.027705    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28df0487_be29_4efe_99d6_94b6166fa1cc.slice/crio-c9a3174df1cc79b05dd7f5c8155617e78cd882eb3e1f1160357260af660b0861 WatchSource:0}: Error finding container c9a3174df1cc79b05dd7f5c8155617e78cd882eb3e1f1160357260af660b0861: Status 404 returned error can't find the container with id c9a3174df1cc79b05dd7f5c8155617e78cd882eb3e1f1160357260af660b0861
Nov 03 16:15:20.172933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:20.172835    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" event={"ID":"28df0487-be29-4efe-99d6-94b6166fa1cc","Type":"ContainerStarted","Data":"c9a3174df1cc79b05dd7f5c8155617e78cd882eb3e1f1160357260af660b0861"}
Nov 03 16:15:20.174927 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:20.174902    6506 generic.go:356] "Generic (PLEG): container finished" podID="73ae3a99-2617-44be-9b8d-89b102f079d0" containerID="621004a316b82f226ef77048b25b5f9086888a33b5e8333a90aa8ffff08f1a77" exitCode=60
Nov 03 16:15:20.175059 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:20.174988    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk" event={"ID":"73ae3a99-2617-44be-9b8d-89b102f079d0","Type":"ContainerDied","Data":"621004a316b82f226ef77048b25b5f9086888a33b5e8333a90aa8ffff08f1a77"}
Nov 03 16:15:20.175059 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:20.175044    6506 scope.go:122] "RemoveContainer" containerID="4563a2506dd2ded5e75aa7313dd47972a2672db543032695ba3f65dac4e6f718"
Nov 03 16:15:20.175300 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:20.175285    6506 scope.go:122] "RemoveContainer" containerID="621004a316b82f226ef77048b25b5f9086888a33b5e8333a90aa8ffff08f1a77"
Nov 03 16:15:20.175532 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:15:20.175512    6506 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895375-rhqkk_odh-ai-gateway-infra(73ae3a99-2617-44be-9b8d-89b102f079d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk" podUID="73ae3a99-2617-44be-9b8d-89b102f079d0"
Nov 03 16:15:21.181699 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:21.181668    6506 scope.go:122] "RemoveContainer" containerID="621004a316b82f226ef77048b25b5f9086888a33b5e8333a90aa8ffff08f1a77"
Nov 03 16:15:21.182181 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:15:21.181970    6506 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895375-rhqkk_odh-ai-gateway-infra(73ae3a99-2617-44be-9b8d-89b102f079d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk" podUID="73ae3a99-2617-44be-9b8d-89b102f079d0"
Nov 03 16:15:21.182999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:21.182973    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" event={"ID":"28df0487-be29-4efe-99d6-94b6166fa1cc","Type":"ContainerStarted","Data":"7a2cc3926c456de359b4246af3899dea3415625e419e0ec3c049e26696aded90"}
Nov 03 16:15:21.226818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:21.224483    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podStartSLOduration=1.89916994 podStartE2EDuration="2.224461379s" totalImagesPullingTime="325.291439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:21.222827393 +0000 UTC m=+735.236225537" watchObservedRunningTime="2026-11-03 16:15:21.224461379 +0000 UTC m=+735.237859526"
Nov 03 16:15:21.736649 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:21.736607    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:22.126458 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:22.126421    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:15:22.959247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:22.959215    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"]
Nov 03 16:15:22.962849 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:22.962825    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:22.970989 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:22.970967    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 03 16:15:22.975901 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:22.975880    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"]
Nov 03 16:15:23.009966 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.009940    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vddkt\" (UniqueName: \"kubernetes.io/projected/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-kube-api-access-vddkt\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.010092 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.009988    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.010152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.010108    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.010152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.010146    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.010243 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.010189    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.010243 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.010216    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.077938 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.077911    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:15:23.111352 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.111320    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.111352 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.111359    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.111558 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.111489    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vddkt\" (UniqueName: \"kubernetes.io/projected/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-kube-api-access-vddkt\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.111558 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.111544    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.111647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.111614    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.111692 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.111653    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.112046 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.112023    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.112165 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.112031    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.112165 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.112125    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.114352 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.114327    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.114515 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.114498    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.124004 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.123986    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vddkt\" (UniqueName: \"kubernetes.io/projected/b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45-kube-api-access-vddkt\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lqsws\" (UID: \"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.276015 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.275987    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:23.554866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.554840    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"]
Nov 03 16:15:23.556626 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:15:23.556603    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81f4c2b_2a82_4d5d_b12e_58ada0bfdb45.slice/crio-9a5a93ac390869b90139eedcfdb859639f34107dab8abbb46e7cec05b623275d WatchSource:0}: Error finding container 9a5a93ac390869b90139eedcfdb859639f34107dab8abbb46e7cec05b623275d: Status 404 returned error can't find the container with id 9a5a93ac390869b90139eedcfdb859639f34107dab8abbb46e7cec05b623275d
Nov 03 16:15:23.759329 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.759294    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"]
Nov 03 16:15:23.764193 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.764170    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.777269 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.777244    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 16:15:23.779007 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.778983    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"]
Nov 03 16:15:23.818099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.818015    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/993750b4-6f00-4cb4-b14e-d1ef7c7249df-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.818099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.818046    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/993750b4-6f00-4cb4-b14e-d1ef7c7249df-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.818099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.818088    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993750b4-6f00-4cb4-b14e-d1ef7c7249df-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.818321 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.818227    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr5p5\" (UniqueName: \"kubernetes.io/projected/993750b4-6f00-4cb4-b14e-d1ef7c7249df-kube-api-access-jr5p5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.818321 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.818282    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/993750b4-6f00-4cb4-b14e-d1ef7c7249df-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.818321 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.818317    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/993750b4-6f00-4cb4-b14e-d1ef7c7249df-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.869527 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.869496    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:15:23.919692 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.919662    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/993750b4-6f00-4cb4-b14e-d1ef7c7249df-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.919850 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.919706    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/993750b4-6f00-4cb4-b14e-d1ef7c7249df-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.919850 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.919766    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993750b4-6f00-4cb4-b14e-d1ef7c7249df-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.919850 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.919836    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr5p5\" (UniqueName: \"kubernetes.io/projected/993750b4-6f00-4cb4-b14e-d1ef7c7249df-kube-api-access-jr5p5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.920027 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.919902    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/993750b4-6f00-4cb4-b14e-d1ef7c7249df-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.920027 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.919935    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/993750b4-6f00-4cb4-b14e-d1ef7c7249df-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.920156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.920055    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/993750b4-6f00-4cb4-b14e-d1ef7c7249df-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.920224 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.920199    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/993750b4-6f00-4cb4-b14e-d1ef7c7249df-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.920308 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.920283    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/993750b4-6f00-4cb4-b14e-d1ef7c7249df-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.922636 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.922614    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/993750b4-6f00-4cb4-b14e-d1ef7c7249df-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.922914 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.922891    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993750b4-6f00-4cb4-b14e-d1ef7c7249df-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:23.938252 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:23.938225    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr5p5\" (UniqueName: \"kubernetes.io/projected/993750b4-6f00-4cb4-b14e-d1ef7c7249df-kube-api-access-jr5p5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9cw7q\" (UID: \"993750b4-6f00-4cb4-b14e-d1ef7c7249df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:24.080046 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:24.079957    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:24.200024 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:24.199976    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws" event={"ID":"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45","Type":"ContainerStarted","Data":"3fe6105194ba4410f54db2b2b64750bc8da1bd59d70c04cd7cfa3b85a1d97278"}
Nov 03 16:15:24.200207 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:24.200046    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws" event={"ID":"b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45","Type":"ContainerStarted","Data":"9a5a93ac390869b90139eedcfdb859639f34107dab8abbb46e7cec05b623275d"}
Nov 03 16:15:24.225113 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:24.225050    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws" podStartSLOduration=1.887677598 podStartE2EDuration="2.225036953s" totalImagesPullingTime="337.359355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:24.222302481 +0000 UTC m=+738.235700630" watchObservedRunningTime="2026-11-03 16:15:24.225036953 +0000 UTC m=+738.238435099"
Nov 03 16:15:24.276352 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:24.276304    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:24.389119 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:24.377712    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"]
Nov 03 16:15:25.213497 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:25.213404    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q" event={"ID":"993750b4-6f00-4cb4-b14e-d1ef7c7249df","Type":"ContainerStarted","Data":"7470c16669016eeb2e03f3ca530ed10cbbfb34c0cc90fda62a45cdeac85e2553"}
Nov 03 16:15:25.214047 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:25.213498    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q" event={"ID":"993750b4-6f00-4cb4-b14e-d1ef7c7249df","Type":"ContainerStarted","Data":"ae1b02f165d95a980c2351fa5a86834865cc0c18044e54a29ff8a9a4fbbbffb8"}
Nov 03 16:15:25.234718 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:25.234665    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q" podStartSLOduration=1.8220252700000001 podStartE2EDuration="2.234648259s" totalImagesPullingTime="412.622989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:25.233968916 +0000 UTC m=+739.247367063" watchObservedRunningTime="2026-11-03 16:15:25.234648259 +0000 UTC m=+739.248046405"
Nov 03 16:15:25.325673 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:25.325630    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:25.338541 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:25.338517    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:26.080874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:26.080843    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:26.230173 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:26.230139    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr"
Nov 03 16:15:28.795403 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:15:28.795367    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:15:29.736519 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:29.736483    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:29.748819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:29.748776    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:30.254098 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:30.254068    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:15:31.776580 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:31.776547    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:15:31.949706 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:31.949671    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:15:32.097099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:32.097021    6506 scope.go:122] "RemoveContainer" containerID="621004a316b82f226ef77048b25b5f9086888a33b5e8333a90aa8ffff08f1a77"
Nov 03 16:15:33.260283 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:33.260191    6506 generic.go:356] "Generic (PLEG): container finished" podID="73ae3a99-2617-44be-9b8d-89b102f079d0" containerID="7dbdbda8fc3925b18aef39e11f7dc388c1c993ac1c6ccbb3407bb4ec5e23086a" exitCode=60
Nov 03 16:15:33.260283 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:33.260271    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk" event={"ID":"73ae3a99-2617-44be-9b8d-89b102f079d0","Type":"ContainerDied","Data":"7dbdbda8fc3925b18aef39e11f7dc388c1c993ac1c6ccbb3407bb4ec5e23086a"}
Nov 03 16:15:33.260782 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:33.260334    6506 scope.go:122] "RemoveContainer" containerID="621004a316b82f226ef77048b25b5f9086888a33b5e8333a90aa8ffff08f1a77"
Nov 03 16:15:33.260782 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:33.260724    6506 scope.go:122] "RemoveContainer" containerID="7dbdbda8fc3925b18aef39e11f7dc388c1c993ac1c6ccbb3407bb4ec5e23086a"
Nov 03 16:15:33.261010 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:15:33.260989    6506 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895375-rhqkk_odh-ai-gateway-infra(73ae3a99-2617-44be-9b8d-89b102f079d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk" podUID="73ae3a99-2617-44be-9b8d-89b102f079d0"
Nov 03 16:15:33.276440 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:33.276424    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:33.289151 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:33.289131    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:34.080998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:34.080968    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:34.093249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:34.093226    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:34.277861 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:34.277832    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9cw7q"
Nov 03 16:15:34.278790 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:34.278773    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lqsws"
Nov 03 16:15:34.292746 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:34.292719    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"]
Nov 03 16:15:34.437125 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:34.437095    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"
Nov 03 16:15:34.513131 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:34.513101    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ae3a99-2617-44be-9b8d-89b102f079d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ae3a99-2617-44be-9b8d-89b102f079d0-maas-api-service-ca\") pod \"73ae3a99-2617-44be-9b8d-89b102f079d0\" (UID: \"73ae3a99-2617-44be-9b8d-89b102f079d0\") "
Nov 03 16:15:34.513293 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:34.513233    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ae3a99-2617-44be-9b8d-89b102f079d0-kube-api-access-2fmgr\" (UniqueName: \"kubernetes.io/projected/73ae3a99-2617-44be-9b8d-89b102f079d0-kube-api-access-2fmgr\") pod \"73ae3a99-2617-44be-9b8d-89b102f079d0\" (UID: \"73ae3a99-2617-44be-9b8d-89b102f079d0\") "
Nov 03 16:15:34.513469 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:34.513447    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ae3a99-2617-44be-9b8d-89b102f079d0-maas-api-service-ca" pod "73ae3a99-2617-44be-9b8d-89b102f079d0" (UID: "73ae3a99-2617-44be-9b8d-89b102f079d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:15:34.515532 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:34.515509    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ae3a99-2617-44be-9b8d-89b102f079d0-kube-api-access-2fmgr" pod "73ae3a99-2617-44be-9b8d-89b102f079d0" (UID: "73ae3a99-2617-44be-9b8d-89b102f079d0"). InnerVolumeSpecName "kube-api-access-2fmgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:15:34.614559 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:34.614491    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ae3a99-2617-44be-9b8d-89b102f079d0-maas-api-service-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:15:34.614559 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:34.614516    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fmgr\" (UniqueName: \"kubernetes.io/projected/73ae3a99-2617-44be-9b8d-89b102f079d0-kube-api-access-2fmgr\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:15:35.273860 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:35.273818    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"
Nov 03 16:15:35.273860 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:35.273830    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk" event={"ID":"73ae3a99-2617-44be-9b8d-89b102f079d0","Type":"ContainerDied","Data":"7a5ec00a9a83cb8565565d418dff033c2feef6231fb6b7b5bbba1719c77d1f96"}
Nov 03 16:15:35.274109 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:35.273897    6506 scope.go:122] "RemoveContainer" containerID="7dbdbda8fc3925b18aef39e11f7dc388c1c993ac1c6ccbb3407bb4ec5e23086a"
Nov 03 16:15:35.297233 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:35.297203    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"]
Nov 03 16:15:35.302812 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:35.302775    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895375-rhqkk"]
Nov 03 16:15:36.092161 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.092124    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"]
Nov 03 16:15:36.092658 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.092636    6506 cpu_manager.go:414] "containerMap: removing container" podUID="73ae3a99-2617-44be-9b8d-89b102f079d0" containerName="cleanup"
Nov 03 16:15:36.092658 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.092657    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ae3a99-2617-44be-9b8d-89b102f079d0" containerName="cleanup"
Nov 03 16:15:36.092895 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.092697    6506 cpu_manager.go:414] "containerMap: removing container" podUID="73ae3a99-2617-44be-9b8d-89b102f079d0" containerName="cleanup"
Nov 03 16:15:36.092895 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.092704    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ae3a99-2617-44be-9b8d-89b102f079d0" containerName="cleanup"
Nov 03 16:15:36.092895 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.092818    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ae3a99-2617-44be-9b8d-89b102f079d0" containerName="cleanup"
Nov 03 16:15:36.092895 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.092828    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ae3a99-2617-44be-9b8d-89b102f079d0" containerName="cleanup"
Nov 03 16:15:36.098437 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.098411    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.105135 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.105111    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ae3a99-2617-44be-9b8d-89b102f079d0" path="/var/lib/kubelet/pods/73ae3a99-2617-44be-9b8d-89b102f079d0/volumes"
Nov 03 16:15:36.112676 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.112653    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"]
Nov 03 16:15:36.229108 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.229051    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqglf\" (UniqueName: \"kubernetes.io/projected/13e172cb-f664-4da0-a922-08b5a0eb29d6-kube-api-access-sqglf\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.229108 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.229115    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.229380 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.229217    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.229380 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.229248    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e172cb-f664-4da0-a922-08b5a0eb29d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.229380 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.229270    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.229475 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.229445    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.229511 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.229492    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.229570 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.229551    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.229616 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.229603    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.330314 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.330283    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.330759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.330324    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.330759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.330347    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.330759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.330478    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.330759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.330557    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqglf\" (UniqueName: \"kubernetes.io/projected/13e172cb-f664-4da0-a922-08b5a0eb29d6-kube-api-access-sqglf\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.330759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.330589    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.330759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.330651    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.330759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.330678    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e172cb-f664-4da0-a922-08b5a0eb29d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.330759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.330708    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.330759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.330756    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.331201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.330766    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.331201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.330907    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.331201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.331075    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.331372 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.331353    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e172cb-f664-4da0-a922-08b5a0eb29d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.332905 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.332885    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.333596 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.333578    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.343784 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.343728    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.344106 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.344088    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqglf\" (UniqueName: \"kubernetes.io/projected/13e172cb-f664-4da0-a922-08b5a0eb29d6-kube-api-access-sqglf\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qqwsb\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.413950 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.413922    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:36.699421 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.699399    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"]
Nov 03 16:15:36.701619 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:15:36.701589    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e172cb_f664_4da0_a922_08b5a0eb29d6.slice/crio-65972e20bbe61f794441111bafae8ed3225e3c436fe9890cedd2c764b73287e3 WatchSource:0}: Error finding container 65972e20bbe61f794441111bafae8ed3225e3c436fe9890cedd2c764b73287e3: Status 404 returned error can't find the container with id 65972e20bbe61f794441111bafae8ed3225e3c436fe9890cedd2c764b73287e3
Nov 03 16:15:36.703771 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.703739    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:15:36.703860 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.703841    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:15:36.703906 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:36.703885    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:15:37.285935 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:37.285902    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" event={"ID":"13e172cb-f664-4da0-a922-08b5a0eb29d6","Type":"ContainerStarted","Data":"2ca18f7e3ca5b61c9d444d0e44c38cbade52bd818c7123d2e0fec5f33688cac6"}
Nov 03 16:15:37.286118 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:37.285968    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" event={"ID":"13e172cb-f664-4da0-a922-08b5a0eb29d6","Type":"ContainerStarted","Data":"65972e20bbe61f794441111bafae8ed3225e3c436fe9890cedd2c764b73287e3"}
Nov 03 16:15:37.425830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:37.423974    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:37.425830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:37.425770    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 16:15:37.426388 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:37.425849    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" podUID="13e172cb-f664-4da0-a922-08b5a0eb29d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 16:15:38.425823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:38.422195    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 16:15:38.425823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:38.422271    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" podUID="13e172cb-f664-4da0-a922-08b5a0eb29d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 16:15:39.422215 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:39.422178    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 16:15:39.422473 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:39.422444    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" podUID="13e172cb-f664-4da0-a922-08b5a0eb29d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 16:15:40.427878 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.427196    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 16:15:40.427878 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.427260    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" podUID="13e172cb-f664-4da0-a922-08b5a0eb29d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 16:15:40.533819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.533209    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" podStartSLOduration=4.533187242 podStartE2EDuration="4.533187242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:37.31405459 +0000 UTC m=+751.327452736" watchObservedRunningTime="2026-11-03 16:15:40.533187242 +0000 UTC m=+754.546585388"
Nov 03 16:15:40.534021 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.533941    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59fc67b59c-ngbw5"]
Nov 03 16:15:40.534487 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.534461    6506 cpu_manager.go:414] "containerMap: removing container" podUID="73ae3a99-2617-44be-9b8d-89b102f079d0" containerName="cleanup"
Nov 03 16:15:40.534487 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.534489    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ae3a99-2617-44be-9b8d-89b102f079d0" containerName="cleanup"
Nov 03 16:15:40.534688 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.534596    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ae3a99-2617-44be-9b8d-89b102f079d0" containerName="cleanup"
Nov 03 16:15:40.537776 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.537755    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59fc67b59c-ngbw5"
Nov 03 16:15:40.545755 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.545732    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59fc67b59c-ngbw5"]
Nov 03 16:15:40.571348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.571324    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgqb2\" (UniqueName: \"kubernetes.io/projected/41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e-kube-api-access-vgqb2\") pod \"authorino-59fc67b59c-ngbw5\" (UID: \"41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e\") " pod="kuadrant-system/authorino-59fc67b59c-ngbw5"
Nov 03 16:15:40.571461 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.571388    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e-tls-cert\") pod \"authorino-59fc67b59c-ngbw5\" (UID: \"41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e\") " pod="kuadrant-system/authorino-59fc67b59c-ngbw5"
Nov 03 16:15:40.672787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.672749    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgqb2\" (UniqueName: \"kubernetes.io/projected/41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e-kube-api-access-vgqb2\") pod \"authorino-59fc67b59c-ngbw5\" (UID: \"41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e\") " pod="kuadrant-system/authorino-59fc67b59c-ngbw5"
Nov 03 16:15:40.672988 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.672829    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e-tls-cert\") pod \"authorino-59fc67b59c-ngbw5\" (UID: \"41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e\") " pod="kuadrant-system/authorino-59fc67b59c-ngbw5"
Nov 03 16:15:40.675342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.675324    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e-tls-cert\") pod \"authorino-59fc67b59c-ngbw5\" (UID: \"41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e\") " pod="kuadrant-system/authorino-59fc67b59c-ngbw5"
Nov 03 16:15:40.685366 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.685309    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgqb2\" (UniqueName: \"kubernetes.io/projected/41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e-kube-api-access-vgqb2\") pod \"authorino-59fc67b59c-ngbw5\" (UID: \"41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e\") " pod="kuadrant-system/authorino-59fc67b59c-ngbw5"
Nov 03 16:15:40.850758 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:40.850716    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59fc67b59c-ngbw5"
Nov 03 16:15:41.133641 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:41.133506    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59fc67b59c-ngbw5"]
Nov 03 16:15:41.136584 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:15:41.136553    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41bdf1a9_3a7b_47fd_b2a6_884348cf7f7e.slice/crio-f3846c5cf9057a424ff3f0b06b009b064072a3e16f624c6b5bf9dcebcc3892d0 WatchSource:0}: Error finding container f3846c5cf9057a424ff3f0b06b009b064072a3e16f624c6b5bf9dcebcc3892d0: Status 404 returned error can't find the container with id f3846c5cf9057a424ff3f0b06b009b064072a3e16f624c6b5bf9dcebcc3892d0
Nov 03 16:15:41.138436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:41.138416    6506 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:15:41.306404 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:41.306369    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59fc67b59c-ngbw5" event={"ID":"41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e","Type":"ContainerStarted","Data":"f3846c5cf9057a424ff3f0b06b009b064072a3e16f624c6b5bf9dcebcc3892d0"}
Nov 03 16:15:41.416241 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:41.416153    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 16:15:41.416477 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:41.416450    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" podUID="13e172cb-f664-4da0-a922-08b5a0eb29d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 16:15:42.338259 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:42.338209    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59fc67b59c-ngbw5" event={"ID":"41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e","Type":"ContainerStarted","Data":"4ce7b9f1a5097b554b24e33fcff0471292e2da61f61fb5b1c32d2a2183ed7e76"}
Nov 03 16:15:42.362726 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:42.361160    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-59fc67b59c-ngbw5" podStartSLOduration=1.584604252 podStartE2EDuration="2.361140053s" totalImagesPullingTime="776.535801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:42.359361638 +0000 UTC m=+756.372759786" watchObservedRunningTime="2026-11-03 16:15:42.361140053 +0000 UTC m=+756.374538205"
Nov 03 16:15:42.389790 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:42.389764    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66467df5bb-hg9qx"]
Nov 03 16:15:42.390351 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:42.390144    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-66467df5bb-hg9qx" podUID="f1812e02-c0db-4ea6-9382-e0331ba89300" containerName="authorino" containerID="cri-o://d7b809b5264d308717a5ba88b6902b03f521f701abb97b069568d5e5b1836bff" gracePeriod=30
Nov 03 16:15:42.437818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:42.435233    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 16:15:42.437818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:42.435286    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" podUID="13e172cb-f664-4da0-a922-08b5a0eb29d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 16:15:42.658688 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:42.658662    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66467df5bb-hg9qx"
Nov 03 16:15:42.792376 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:42.792330    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1812e02-c0db-4ea6-9382-e0331ba89300-kube-api-access-n64ts\" (UniqueName: \"kubernetes.io/projected/f1812e02-c0db-4ea6-9382-e0331ba89300-kube-api-access-n64ts\") pod \"f1812e02-c0db-4ea6-9382-e0331ba89300\" (UID: \"f1812e02-c0db-4ea6-9382-e0331ba89300\") "
Nov 03 16:15:42.792376 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:42.792384    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1812e02-c0db-4ea6-9382-e0331ba89300-tls-cert\" (UniqueName: \"kubernetes.io/secret/f1812e02-c0db-4ea6-9382-e0331ba89300-tls-cert\") pod \"f1812e02-c0db-4ea6-9382-e0331ba89300\" (UID: \"f1812e02-c0db-4ea6-9382-e0331ba89300\") "
Nov 03 16:15:42.795869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:42.795755    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1812e02-c0db-4ea6-9382-e0331ba89300-kube-api-access-n64ts" pod "f1812e02-c0db-4ea6-9382-e0331ba89300" (UID: "f1812e02-c0db-4ea6-9382-e0331ba89300"). InnerVolumeSpecName "kube-api-access-n64ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:15:42.806166 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:42.806132    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1812e02-c0db-4ea6-9382-e0331ba89300-tls-cert" pod "f1812e02-c0db-4ea6-9382-e0331ba89300" (UID: "f1812e02-c0db-4ea6-9382-e0331ba89300"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:15:42.893768 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:42.893729    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n64ts\" (UniqueName: \"kubernetes.io/projected/f1812e02-c0db-4ea6-9382-e0331ba89300-kube-api-access-n64ts\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:15:42.893768 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:42.893762    6506 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1812e02-c0db-4ea6-9382-e0331ba89300-tls-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:15:43.057325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:43.057289    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:15:43.235656 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:43.235562    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:15:43.344119 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:43.344074    6506 generic.go:356] "Generic (PLEG): container finished" podID="f1812e02-c0db-4ea6-9382-e0331ba89300" containerID="d7b809b5264d308717a5ba88b6902b03f521f701abb97b069568d5e5b1836bff" exitCode=0
Nov 03 16:15:43.344570 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:43.344129    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66467df5bb-hg9qx"
Nov 03 16:15:43.344570 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:43.344183    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66467df5bb-hg9qx" event={"ID":"f1812e02-c0db-4ea6-9382-e0331ba89300","Type":"ContainerDied","Data":"d7b809b5264d308717a5ba88b6902b03f521f701abb97b069568d5e5b1836bff"}
Nov 03 16:15:43.344570 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:43.344235    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66467df5bb-hg9qx" event={"ID":"f1812e02-c0db-4ea6-9382-e0331ba89300","Type":"ContainerDied","Data":"6d1e5f979a8d8f09de370911e36137aec233967f3334a617f2fad8ee29b99b23"}
Nov 03 16:15:43.344570 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:43.344265    6506 scope.go:122] "RemoveContainer" containerID="d7b809b5264d308717a5ba88b6902b03f521f701abb97b069568d5e5b1836bff"
Nov 03 16:15:43.354230 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:43.354211    6506 scope.go:122] "RemoveContainer" containerID="d7b809b5264d308717a5ba88b6902b03f521f701abb97b069568d5e5b1836bff"
Nov 03 16:15:43.354499 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:15:43.354477    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b809b5264d308717a5ba88b6902b03f521f701abb97b069568d5e5b1836bff\": container with ID starting with d7b809b5264d308717a5ba88b6902b03f521f701abb97b069568d5e5b1836bff not found: ID does not exist" containerID="d7b809b5264d308717a5ba88b6902b03f521f701abb97b069568d5e5b1836bff"
Nov 03 16:15:43.354568 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:43.354513    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b809b5264d308717a5ba88b6902b03f521f701abb97b069568d5e5b1836bff"} err="failed to get container status \"d7b809b5264d308717a5ba88b6902b03f521f701abb97b069568d5e5b1836bff\": rpc error: code = NotFound desc = could not find container \"d7b809b5264d308717a5ba88b6902b03f521f701abb97b069568d5e5b1836bff\": container with ID starting with d7b809b5264d308717a5ba88b6902b03f521f701abb97b069568d5e5b1836bff not found: ID does not exist"
Nov 03 16:15:43.372326 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:43.372301    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66467df5bb-hg9qx"]
Nov 03 16:15:43.376611 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:43.376590    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-66467df5bb-hg9qx"]
Nov 03 16:15:43.414840 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:43.414785    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 16:15:43.414999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:43.414869    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" podUID="13e172cb-f664-4da0-a922-08b5a0eb29d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 16:15:44.102955 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:44.102925    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1812e02-c0db-4ea6-9382-e0331ba89300" path="/var/lib/kubelet/pods/f1812e02-c0db-4ea6-9382-e0331ba89300/volumes"
Nov 03 16:15:45.095003 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:45.094968    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:45.095737 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:45.095712    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:45.096990 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:45.096969    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:15:47.685817 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:47.682781    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:15:47.842138 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:47.842095    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:15:51.133318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.133285    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"]
Nov 03 16:15:51.133855 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.133656    6506 cpu_manager.go:414] "containerMap: removing container" podUID="f1812e02-c0db-4ea6-9382-e0331ba89300" containerName="authorino"
Nov 03 16:15:51.133855 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.133668    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1812e02-c0db-4ea6-9382-e0331ba89300" containerName="authorino"
Nov 03 16:15:51.133855 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.133750    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1812e02-c0db-4ea6-9382-e0331ba89300" containerName="authorino"
Nov 03 16:15:51.136645 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.136623    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.161107 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.161049    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"]
Nov 03 16:15:51.282573 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.282537    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.282727 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.282582    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02798929-b745-48ba-8db3-31b20be5367b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.282727 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.282665    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.282727 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.282695    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02798929-b745-48ba-8db3-31b20be5367b-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.282727 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.282714    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb9wg\" (UniqueName: \"kubernetes.io/projected/02798929-b745-48ba-8db3-31b20be5367b-kube-api-access-vb9wg\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.282936 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.282763    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.282936 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.282845    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.282936 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.282863    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02798929-b745-48ba-8db3-31b20be5367b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.282936 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.282884    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.383455 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.383382    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.383455 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.383427    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02798929-b745-48ba-8db3-31b20be5367b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.383673 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.383564    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.383673 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.383604    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02798929-b745-48ba-8db3-31b20be5367b-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.383673 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.383631    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb9wg\" (UniqueName: \"kubernetes.io/projected/02798929-b745-48ba-8db3-31b20be5367b-kube-api-access-vb9wg\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.383673 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.383662    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.383934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.383698    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.383934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.383723    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02798929-b745-48ba-8db3-31b20be5367b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.383934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.383842    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.383934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.383845    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.383934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.383929    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.384212 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.384105    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.384212 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.384191    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02798929-b745-48ba-8db3-31b20be5367b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.384287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.384213    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.386017 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.385998    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.386350 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.386335    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02798929-b745-48ba-8db3-31b20be5367b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.396655 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.396630    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb9wg\" (UniqueName: \"kubernetes.io/projected/02798929-b745-48ba-8db3-31b20be5367b-kube-api-access-vb9wg\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.396756 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.396714    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02798929-b745-48ba-8db3-31b20be5367b-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8hmf2\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.453951 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.453927    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:51.737128 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.737106    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"]
Nov 03 16:15:51.739324 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:15:51.739293    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02798929_b745_48ba_8db3_31b20be5367b.slice/crio-994f4332780eeef019151cbd9d4fcd2daaa3e99bbc733159d19425523ba52d86 WatchSource:0}: Error finding container 994f4332780eeef019151cbd9d4fcd2daaa3e99bbc733159d19425523ba52d86: Status 404 returned error can't find the container with id 994f4332780eeef019151cbd9d4fcd2daaa3e99bbc733159d19425523ba52d86
Nov 03 16:15:51.741824 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.741772    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:15:51.741911 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.741872    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:15:51.741981 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:51.741917    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:15:52.398721 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:52.398685    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2" event={"ID":"02798929-b745-48ba-8db3-31b20be5367b","Type":"ContainerStarted","Data":"b7e9fe6c0f5024c5c5bfc55f16fd8a664c7056eba751680a581a5bd410bfbc4e"}
Nov 03 16:15:52.399106 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:52.398735    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2" event={"ID":"02798929-b745-48ba-8db3-31b20be5367b","Type":"ContainerStarted","Data":"994f4332780eeef019151cbd9d4fcd2daaa3e99bbc733159d19425523ba52d86"}
Nov 03 16:15:52.435196 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:52.435144    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2" podStartSLOduration=1.4351260940000001 podStartE2EDuration="1.435126094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:52.432321001 +0000 UTC m=+766.445719153" watchObservedRunningTime="2026-11-03 16:15:52.435126094 +0000 UTC m=+766.448524239"
Nov 03 16:15:52.454888 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:52.454854    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:52.456259 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:52.456234    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 16:15:52.456365 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:52.456285    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2" podUID="02798929-b745-48ba-8db3-31b20be5367b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 16:15:53.454541 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:53.454506    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 16:15:53.454939 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:53.454559    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2" podUID="02798929-b745-48ba-8db3-31b20be5367b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 16:15:54.455104 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:54.455071    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 16:15:54.455515 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:54.455129    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2" podUID="02798929-b745-48ba-8db3-31b20be5367b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 16:15:55.454851 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:55.454788    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 16:15:55.455044 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:55.454876    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2" podUID="02798929-b745-48ba-8db3-31b20be5367b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 16:15:56.454835 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:56.454782    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 16:15:56.455225 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:56.454866    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2" podUID="02798929-b745-48ba-8db3-31b20be5367b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 16:15:57.455276 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:57.455240    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 16:15:57.455668 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:57.455294    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2" podUID="02798929-b745-48ba-8db3-31b20be5367b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 16:15:58.455113 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:58.455070    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 16:15:58.455320 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:58.455146    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2" podUID="02798929-b745-48ba-8db3-31b20be5367b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 16:15:58.795497 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:15:58.795457    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:15:59.923084 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:59.923047    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:59.923566 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:59.923411    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:15:59.924154 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:15:59.924136    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:16:06.148777 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.148744    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"]
Nov 03 16:16:06.154181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.154157    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.165266 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.165242    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"]
Nov 03 16:16:06.261370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.261337    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.261519 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.261380    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.261519 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.261405    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.261601 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.261546    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.261636 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.261599    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drg4n\" (UniqueName: \"kubernetes.io/projected/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-kube-api-access-drg4n\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.261636 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.261631    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.261713 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.261666    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.261757 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.261736    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.261838 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.261762    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.362247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.362217    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.362405 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.362268    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.362405 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.362380    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.362507 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.362433    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.362507 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.362453    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.362507 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.362483    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.362689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.362506    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.362689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.362533    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.362689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.362575    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drg4n\" (UniqueName: \"kubernetes.io/projected/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-kube-api-access-drg4n\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.362923 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.362902    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.363070 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.362977    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.363168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.363005    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.363168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.363013    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.363168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.363153    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.364919 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.364896    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.365046 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.364939    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.375954 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.375930    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.376061 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.376009    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drg4n\" (UniqueName: \"kubernetes.io/projected/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-kube-api-access-drg4n\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bs7fr\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.471750 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.471675    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:06.755617 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.755591    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"]
Nov 03 16:16:06.757228 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:16:06.757201    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1ab2e2_737e_490f_b820_8ce3be90c1b5.slice/crio-764e57f78891c745d2d47b8218970c5f207773013c5db2b244ac79b2059c59e0 WatchSource:0}: Error finding container 764e57f78891c745d2d47b8218970c5f207773013c5db2b244ac79b2059c59e0: Status 404 returned error can't find the container with id 764e57f78891c745d2d47b8218970c5f207773013c5db2b244ac79b2059c59e0
Nov 03 16:16:06.759338 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.759310    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:16:06.759424 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.759379    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:16:06.759490 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:06.759422    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:16:07.505665 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:07.505632    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr" event={"ID":"bb1ab2e2-737e-490f-b820-8ce3be90c1b5","Type":"ContainerStarted","Data":"84cb5cddef36a6812b2953b7ab30ca5587fa6b40a1f323fb460f74c51157bf30"}
Nov 03 16:16:07.506174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:07.505690    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr" event={"ID":"bb1ab2e2-737e-490f-b820-8ce3be90c1b5","Type":"ContainerStarted","Data":"764e57f78891c745d2d47b8218970c5f207773013c5db2b244ac79b2059c59e0"}
Nov 03 16:16:07.551494 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:07.551443    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr" podStartSLOduration=1.551428647 podStartE2EDuration="1.551428647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:07.549065485 +0000 UTC m=+781.562463622" watchObservedRunningTime="2026-11-03 16:16:07.551428647 +0000 UTC m=+781.564826792"
Nov 03 16:16:08.472461 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:08.472422    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:08.473753 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:08.473727    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 16:16:08.473900 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:08.473774    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr" podUID="bb1ab2e2-737e-490f-b820-8ce3be90c1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 16:16:09.064009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:09.063979    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:09.472965 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:09.472889    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 16:16:09.472965 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:09.472942    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr" podUID="bb1ab2e2-737e-490f-b820-8ce3be90c1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 16:16:10.472546 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:10.472508    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 16:16:10.472995 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:10.472565    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr" podUID="bb1ab2e2-737e-490f-b820-8ce3be90c1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 16:16:11.472889 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:11.472853    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 16:16:11.473291 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:11.472913    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr" podUID="bb1ab2e2-737e-490f-b820-8ce3be90c1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 16:16:12.433128 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:12.433057    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:12.472287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:12.472256    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 16:16:12.472403 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:12.472319    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr" podUID="bb1ab2e2-737e-490f-b820-8ce3be90c1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 16:16:12.949195 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:12.949162    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:13.472549 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:13.472505    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 16:16:13.472719 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:13.472578    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr" podUID="bb1ab2e2-737e-490f-b820-8ce3be90c1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 16:16:14.557765 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:14.557715    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:14.818574 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:14.818484    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:14.920663 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:14.920633    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:14.921054 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:14.921030    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:14.921775 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:14.921754    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:16:16.042547 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:16.042511    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:16.222807 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:16.222769    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:28.795568 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:16:28.795528    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:16:38.022912 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.022875    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:38.189903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.189872    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:38.426102 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.426027    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"]
Nov 03 16:16:38.430893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.430875    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.439937 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.439916    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b132e3-openshift-default-dockercfg-d5hc7\""
Nov 03 16:16:38.447186 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.447141    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"]
Nov 03 16:16:38.476532 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.476506    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.476661 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.476550    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.476661 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.476579    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.476661 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.476600    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.476661 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.476624    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.476870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.476659    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.476870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.476712    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.476870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.476745    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.476870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.476809    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px495\" (UniqueName: \"kubernetes.io/projected/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-kube-api-access-px495\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.574249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.574210    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"]
Nov 03 16:16:38.574937 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.574917    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:38.577907 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.577887    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.578164 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.578145    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.578246 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.578178    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.578246 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.578210    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.578361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.578245    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.578361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.578295    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.578361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.578335    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.578514 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.578448    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px495\" (UniqueName: \"kubernetes.io/projected/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-kube-api-access-px495\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.578701 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.578674    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.578787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.578703    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.578869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.578752    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.578869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.578820    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.579149 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.578869    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.579149 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.578908    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.579149 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.579043    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.580732 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.580710    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.580842 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.580750    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.584436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.584417    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c09b17-openshift-default-dockercfg-2jjh9\""
Nov 03 16:16:38.596534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.596512    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px495\" (UniqueName: \"kubernetes.io/projected/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-kube-api-access-px495\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.602927 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.602903    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"]
Nov 03 16:16:38.607486 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.607466    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.679603 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.679513    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.679603 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.679550    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcsrb\" (UniqueName: \"kubernetes.io/projected/e0ed082c-356f-45b5-80e5-c70a6f5d055d-kube-api-access-rcsrb\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.679603 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.679580    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.679843 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.679812    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.679906 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.679891    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.679965 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.679931    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.680025 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.679993    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.680164 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.680139    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.680205 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.680185    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.768496 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.768466    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:38.781594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.781562    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.781711 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.781624    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.781711 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.781669    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.781845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.781765    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.781845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.781822    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.781958 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.781869    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.781958 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.781899    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcsrb\" (UniqueName: \"kubernetes.io/projected/e0ed082c-356f-45b5-80e5-c70a6f5d055d-kube-api-access-rcsrb\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.781958 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.781931    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.782105 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.782045    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.782105 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.782054    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.782199 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.782103    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.782402 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.782349    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.782492 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.782428    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.782692 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.782663    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.784051 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.784024    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.784933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.784908    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.799874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.799850    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.800095 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.800077    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcsrb\" (UniqueName: \"kubernetes.io/projected/e0ed082c-356f-45b5-80e5-c70a6f5d055d-kube-api-access-rcsrb\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:38.814184 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.814160    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:38.900009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:38.899980    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:39.000421 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.000391    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:39.058308 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.058279    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"]
Nov 03 16:16:39.062441 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.062420    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.063637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.063614    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"]
Nov 03 16:16:39.065580 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:16:39.065556    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f2a444_a2bf_46ed_9a2d_4a40dd09c537.slice/crio-94a8eee9dcb8a79512a24572a093bcfe0adff24b4b5a257620e7daa146d770bb WatchSource:0}: Error finding container 94a8eee9dcb8a79512a24572a093bcfe0adff24b4b5a257620e7daa146d770bb: Status 404 returned error can't find the container with id 94a8eee9dcb8a79512a24572a093bcfe0adff24b4b5a257620e7daa146d770bb
Nov 03 16:16:39.068135 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.068105    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:16:39.068243 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.068175    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:16:39.068243 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.068217    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:16:39.075661 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.075639    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-afb990-openshift-default-dockercfg-gc64q\""
Nov 03 16:16:39.098467 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.098441    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"]
Nov 03 16:16:39.186148 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.186111    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b9b08d-2d97-4604-890c-10f6305c0108-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.186331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.186154    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.186331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.186278    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzbvq\" (UniqueName: \"kubernetes.io/projected/20b9b08d-2d97-4604-890c-10f6305c0108-kube-api-access-qzbvq\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.186453 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.186334    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.186546 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.186526    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.186609 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.186558    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9b08d-2d97-4604-890c-10f6305c0108-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.186667 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.186604    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.186667 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.186643    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9b08d-2d97-4604-890c-10f6305c0108-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.186778 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.186673    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.224259 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.224236    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"]
Nov 03 16:16:39.227352 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:16:39.227324    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ed082c_356f_45b5_80e5_c70a6f5d055d.slice/crio-c08e16f6662088843697aa6821be69405c4bf0376d589e293135553382f99d1d WatchSource:0}: Error finding container c08e16f6662088843697aa6821be69405c4bf0376d589e293135553382f99d1d: Status 404 returned error can't find the container with id c08e16f6662088843697aa6821be69405c4bf0376d589e293135553382f99d1d
Nov 03 16:16:39.229999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.229968    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:16:39.230089 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.230025    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:16:39.230089 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.230051    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:16:39.247057 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.247029    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:39.288340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.288308    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.288508 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.288368    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9b08d-2d97-4604-890c-10f6305c0108-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.288508 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.288477    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.288647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.288621    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b9b08d-2d97-4604-890c-10f6305c0108-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.288715 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.288656    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.288715 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.288700    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzbvq\" (UniqueName: \"kubernetes.io/projected/20b9b08d-2d97-4604-890c-10f6305c0108-kube-api-access-qzbvq\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.288840 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.288737    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.288840 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.288755    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.288840 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.288831    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.289000 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.288865    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.289000 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.288892    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9b08d-2d97-4604-890c-10f6305c0108-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.289115 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.289000    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.289366 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.289311    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.289998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.289975    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9b08d-2d97-4604-890c-10f6305c0108-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.292403 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.292155    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9b08d-2d97-4604-890c-10f6305c0108-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.292566 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.292429    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.302712 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.302687    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b9b08d-2d97-4604-890c-10f6305c0108-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.304965 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.304943    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzbvq\" (UniqueName: \"kubernetes.io/projected/20b9b08d-2d97-4604-890c-10f6305c0108-kube-api-access-qzbvq\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.379377 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.379335    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:39.408251 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.408180    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:39.569167 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.569121    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:39.667117 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.667091    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"]
Nov 03 16:16:39.668690 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:16:39.668657    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b9b08d_2d97_4604_890c_10f6305c0108.slice/crio-b0f4aae46f31754dd634d4a97c8eccfee23c377326f29624fe61c064a6a77d32 WatchSource:0}: Error finding container b0f4aae46f31754dd634d4a97c8eccfee23c377326f29624fe61c064a6a77d32: Status 404 returned error can't find the container with id b0f4aae46f31754dd634d4a97c8eccfee23c377326f29624fe61c064a6a77d32
Nov 03 16:16:39.671046 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.671017    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:16:39.671121 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.671079    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:16:39.671121 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.671104    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:16:39.686828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.686767    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw" event={"ID":"20b9b08d-2d97-4604-890c-10f6305c0108","Type":"ContainerStarted","Data":"b0f4aae46f31754dd634d4a97c8eccfee23c377326f29624fe61c064a6a77d32"}
Nov 03 16:16:39.696356 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.696320    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5" event={"ID":"e0ed082c-356f-45b5-80e5-c70a6f5d055d","Type":"ContainerStarted","Data":"36ae5eab43b7543917c52fcfb1337e71d5e7c819767dac31f60642adb1f4e97b"}
Nov 03 16:16:39.696475 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.696402    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5" event={"ID":"e0ed082c-356f-45b5-80e5-c70a6f5d055d","Type":"ContainerStarted","Data":"c08e16f6662088843697aa6821be69405c4bf0376d589e293135553382f99d1d"}
Nov 03 16:16:39.698059 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.698036    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt" event={"ID":"15f2a444-a2bf-46ed-9a2d-4a40dd09c537","Type":"ContainerStarted","Data":"cd4df6ddd25a97f54b079e7943d6c6c6791ad373e3e2727e1f3550a0f0c01aab"}
Nov 03 16:16:39.698162 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.698074    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt" event={"ID":"15f2a444-a2bf-46ed-9a2d-4a40dd09c537","Type":"ContainerStarted","Data":"94a8eee9dcb8a79512a24572a093bcfe0adff24b4b5a257620e7daa146d770bb"}
Nov 03 16:16:39.727239 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.727180    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5" podStartSLOduration=1.727167116 podStartE2EDuration="1.727167116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:39.724025752 +0000 UTC m=+813.737423899" watchObservedRunningTime="2026-11-03 16:16:39.727167116 +0000 UTC m=+813.740565261"
Nov 03 16:16:39.750302 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.750254    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt" podStartSLOduration=1.75023757 podStartE2EDuration="1.75023757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:39.747963149 +0000 UTC m=+813.761361297" watchObservedRunningTime="2026-11-03 16:16:39.75023757 +0000 UTC m=+813.763635717"
Nov 03 16:16:39.769438 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.769413    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:39.900961 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:39.900878    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:40.704524 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:40.704487    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw" event={"ID":"20b9b08d-2d97-4604-890c-10f6305c0108","Type":"ContainerStarted","Data":"1551a6edd5f3fcca5c624548874ffe5096f2c67fe1cd90d38c89ec5e8d7f553a"}
Nov 03 16:16:40.734177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:40.734122    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw" podStartSLOduration=1.734105418 podStartE2EDuration="1.734105418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:40.733938935 +0000 UTC m=+814.747337095" watchObservedRunningTime="2026-11-03 16:16:40.734105418 +0000 UTC m=+814.747503566"
Nov 03 16:16:40.773897 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:40.773868    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:40.905404 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:40.905376    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:41.379925 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:41.379896    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:41.385315 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:41.385292    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:41.710393 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:41.710311    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:41.710393 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:41.710366    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:41.710393 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:41.710387    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:41.711119 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:41.711092    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:16:41.711218 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:41.711149    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:16:41.711310 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:41.711292    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:16:42.144447 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:42.144416    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:42.380244 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:42.380193    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:42.594194 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:42.594155    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:42.792935 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:42.792904    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:43.169573 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.169539    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:43.352070 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.352034    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:43.557326 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.557293    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:43.609499 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.609471    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"]
Nov 03 16:16:43.613147 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.613132    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"
Nov 03 16:16:43.631819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.631770    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-b132e3-dockercfg-49x9w\""
Nov 03 16:16:43.635174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.635152    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-b132e3\""
Nov 03 16:16:43.649360 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.649337    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"]
Nov 03 16:16:43.651926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.651896    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"]
Nov 03 16:16:43.659934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.659911    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"
Nov 03 16:16:43.669193 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.669166    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"]
Nov 03 16:16:43.740153 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.740114    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns5kx\" (UniqueName: \"kubernetes.io/projected/ccadaa1e-d59a-4231-ad4c-d8f46e7f4365-kube-api-access-ns5kx\") pod \"payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t\" (UID: \"ccadaa1e-d59a-4231-ad4c-d8f46e7f4365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"
Nov 03 16:16:43.740347 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.740173    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d53d56-ee8a-4465-8a5a-06e9064ab695-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2\" (UID: \"f5d53d56-ee8a-4465-8a5a-06e9064ab695\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"
Nov 03 16:16:43.740422 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.740336    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccadaa1e-d59a-4231-ad4c-d8f46e7f4365-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t\" (UID: \"ccadaa1e-d59a-4231-ad4c-d8f46e7f4365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"
Nov 03 16:16:43.740422 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.740415    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpfn9\" (UniqueName: \"kubernetes.io/projected/f5d53d56-ee8a-4465-8a5a-06e9064ab695-kube-api-access-tpfn9\") pod \"payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2\" (UID: \"f5d53d56-ee8a-4465-8a5a-06e9064ab695\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"
Nov 03 16:16:43.793511 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.793469    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:43.842000 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.841912    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpfn9\" (UniqueName: \"kubernetes.io/projected/f5d53d56-ee8a-4465-8a5a-06e9064ab695-kube-api-access-tpfn9\") pod \"payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2\" (UID: \"f5d53d56-ee8a-4465-8a5a-06e9064ab695\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"
Nov 03 16:16:43.842272 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.842246    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns5kx\" (UniqueName: \"kubernetes.io/projected/ccadaa1e-d59a-4231-ad4c-d8f46e7f4365-kube-api-access-ns5kx\") pod \"payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t\" (UID: \"ccadaa1e-d59a-4231-ad4c-d8f46e7f4365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"
Nov 03 16:16:43.842383 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.842340    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d53d56-ee8a-4465-8a5a-06e9064ab695-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2\" (UID: \"f5d53d56-ee8a-4465-8a5a-06e9064ab695\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"
Nov 03 16:16:43.842472 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.842453    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccadaa1e-d59a-4231-ad4c-d8f46e7f4365-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t\" (UID: \"ccadaa1e-d59a-4231-ad4c-d8f46e7f4365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"
Nov 03 16:16:43.843050 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.843031    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d53d56-ee8a-4465-8a5a-06e9064ab695-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2\" (UID: \"f5d53d56-ee8a-4465-8a5a-06e9064ab695\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"
Nov 03 16:16:43.843134 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.843118    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccadaa1e-d59a-4231-ad4c-d8f46e7f4365-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t\" (UID: \"ccadaa1e-d59a-4231-ad4c-d8f46e7f4365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"
Nov 03 16:16:43.856788 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.856757    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpfn9\" (UniqueName: \"kubernetes.io/projected/f5d53d56-ee8a-4465-8a5a-06e9064ab695-kube-api-access-tpfn9\") pod \"payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2\" (UID: \"f5d53d56-ee8a-4465-8a5a-06e9064ab695\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"
Nov 03 16:16:43.856788 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.856770    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns5kx\" (UniqueName: \"kubernetes.io/projected/ccadaa1e-d59a-4231-ad4c-d8f46e7f4365-kube-api-access-ns5kx\") pod \"payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t\" (UID: \"ccadaa1e-d59a-4231-ad4c-d8f46e7f4365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"
Nov 03 16:16:43.925104 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.925063    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"
Nov 03 16:16:43.981436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:43.981199    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"
Nov 03 16:16:44.205849 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.205824    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"]
Nov 03 16:16:44.206846 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:16:44.206818    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccadaa1e_d59a_4231_ad4c_d8f46e7f4365.slice/crio-3a2fd32ddb3707137d90fa650da157c72c4deb162c08bb9117018120c933a177 WatchSource:0}: Error finding container 3a2fd32ddb3707137d90fa650da157c72c4deb162c08bb9117018120c933a177: Status 404 returned error can't find the container with id 3a2fd32ddb3707137d90fa650da157c72c4deb162c08bb9117018120c933a177
Nov 03 16:16:44.272984 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.272959    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"]
Nov 03 16:16:44.275296 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:16:44.275256    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d53d56_ee8a_4465_8a5a_06e9064ab695.slice/crio-6d37e37ef0946caf88a753638c64f568eec54d050f1126fe06b533b602c78060 WatchSource:0}: Error finding container 6d37e37ef0946caf88a753638c64f568eec54d050f1126fe06b533b602c78060: Status 404 returned error can't find the container with id 6d37e37ef0946caf88a753638c64f568eec54d050f1126fe06b533b602c78060
Nov 03 16:16:44.509432 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.509353    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:44.573911 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.573867    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"]
Nov 03 16:16:44.578759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.578736    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"]
Nov 03 16:16:44.578916 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.578885    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"
Nov 03 16:16:44.582174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.582152    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"
Nov 03 16:16:44.586292 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.586259    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-c09b17\""
Nov 03 16:16:44.599103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.599082    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"]
Nov 03 16:16:44.601644 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.601621    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"]
Nov 03 16:16:44.651187 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.651147    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde0ec03-70f8-47bb-af69-6a56524d425f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n\" (UID: \"dde0ec03-70f8-47bb-af69-6a56524d425f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"
Nov 03 16:16:44.651187 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.651187    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgntb\" (UniqueName: \"kubernetes.io/projected/80ee90cc-3ecf-40be-9d34-511fb0b42721-kube-api-access-fgntb\") pod \"payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g\" (UID: \"80ee90cc-3ecf-40be-9d34-511fb0b42721\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"
Nov 03 16:16:44.651371 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.651321    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ee90cc-3ecf-40be-9d34-511fb0b42721-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g\" (UID: \"80ee90cc-3ecf-40be-9d34-511fb0b42721\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"
Nov 03 16:16:44.651445 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.651431    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqf59\" (UniqueName: \"kubernetes.io/projected/dde0ec03-70f8-47bb-af69-6a56524d425f-kube-api-access-gqf59\") pod \"payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n\" (UID: \"dde0ec03-70f8-47bb-af69-6a56524d425f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"
Nov 03 16:16:44.684287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.684264    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:44.728239 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.728199    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t" event={"ID":"ccadaa1e-d59a-4231-ad4c-d8f46e7f4365","Type":"ContainerStarted","Data":"3a2fd32ddb3707137d90fa650da157c72c4deb162c08bb9117018120c933a177"}
Nov 03 16:16:44.729550 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.729525    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2" event={"ID":"f5d53d56-ee8a-4465-8a5a-06e9064ab695","Type":"ContainerStarted","Data":"6d37e37ef0946caf88a753638c64f568eec54d050f1126fe06b533b602c78060"}
Nov 03 16:16:44.780828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.780610    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqf59\" (UniqueName: \"kubernetes.io/projected/dde0ec03-70f8-47bb-af69-6a56524d425f-kube-api-access-gqf59\") pod \"payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n\" (UID: \"dde0ec03-70f8-47bb-af69-6a56524d425f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"
Nov 03 16:16:44.780828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.780691    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde0ec03-70f8-47bb-af69-6a56524d425f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n\" (UID: \"dde0ec03-70f8-47bb-af69-6a56524d425f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"
Nov 03 16:16:44.780828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.780723    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgntb\" (UniqueName: \"kubernetes.io/projected/80ee90cc-3ecf-40be-9d34-511fb0b42721-kube-api-access-fgntb\") pod \"payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g\" (UID: \"80ee90cc-3ecf-40be-9d34-511fb0b42721\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"
Nov 03 16:16:44.780828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.780779    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ee90cc-3ecf-40be-9d34-511fb0b42721-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g\" (UID: \"80ee90cc-3ecf-40be-9d34-511fb0b42721\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"
Nov 03 16:16:44.785857 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.782254    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde0ec03-70f8-47bb-af69-6a56524d425f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n\" (UID: \"dde0ec03-70f8-47bb-af69-6a56524d425f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"
Nov 03 16:16:44.785857 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.782706    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ee90cc-3ecf-40be-9d34-511fb0b42721-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g\" (UID: \"80ee90cc-3ecf-40be-9d34-511fb0b42721\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"
Nov 03 16:16:44.795129 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.795099    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqf59\" (UniqueName: \"kubernetes.io/projected/dde0ec03-70f8-47bb-af69-6a56524d425f-kube-api-access-gqf59\") pod \"payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n\" (UID: \"dde0ec03-70f8-47bb-af69-6a56524d425f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"
Nov 03 16:16:44.796640 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.796599    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgntb\" (UniqueName: \"kubernetes.io/projected/80ee90cc-3ecf-40be-9d34-511fb0b42721-kube-api-access-fgntb\") pod \"payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g\" (UID: \"80ee90cc-3ecf-40be-9d34-511fb0b42721\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"
Nov 03 16:16:44.886519 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.886433    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:44.893906 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.893879    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"
Nov 03 16:16:44.903030 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:44.903009    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"
Nov 03 16:16:45.209509 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:45.209481    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"]
Nov 03 16:16:45.210832 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:16:45.210789    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ee90cc_3ecf_40be_9d34_511fb0b42721.slice/crio-3ba05e7161afeea8fab860ffe86877747fdea940296070b399f71aff304bdd3e WatchSource:0}: Error finding container 3ba05e7161afeea8fab860ffe86877747fdea940296070b399f71aff304bdd3e: Status 404 returned error can't find the container with id 3ba05e7161afeea8fab860ffe86877747fdea940296070b399f71aff304bdd3e
Nov 03 16:16:45.238437 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:45.238381    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"]
Nov 03 16:16:45.503149 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:45.503069    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:45.750264 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:45.750184    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n" event={"ID":"dde0ec03-70f8-47bb-af69-6a56524d425f","Type":"ContainerStarted","Data":"b4f6a130efc0435a0d3dc7fc35b841e1e91f785e5abb5d291f61131e51777254"}
Nov 03 16:16:45.756519 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:45.756447    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g" event={"ID":"80ee90cc-3ecf-40be-9d34-511fb0b42721","Type":"ContainerStarted","Data":"3ba05e7161afeea8fab860ffe86877747fdea940296070b399f71aff304bdd3e"}
Nov 03 16:16:45.785593 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:45.785528    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:45.991896 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:45.991860    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:46.456204 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.456169    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"]
Nov 03 16:16:46.466869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.466839    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"
Nov 03 16:16:46.475695 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.475671    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-69cedd\""
Nov 03 16:16:46.495450 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.495367    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:46.497245 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.497220    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"]
Nov 03 16:16:46.621942 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.621723    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/381eb05d-c83e-49b3-89b3-935ea62fb4e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c\" (UID: \"381eb05d-c83e-49b3-89b3-935ea62fb4e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"
Nov 03 16:16:46.621942 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.621778    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgznh\" (UniqueName: \"kubernetes.io/projected/381eb05d-c83e-49b3-89b3-935ea62fb4e1-kube-api-access-rgznh\") pod \"payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c\" (UID: \"381eb05d-c83e-49b3-89b3-935ea62fb4e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"
Nov 03 16:16:46.660090 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.659063    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:46.723379 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.723300    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/381eb05d-c83e-49b3-89b3-935ea62fb4e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c\" (UID: \"381eb05d-c83e-49b3-89b3-935ea62fb4e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"
Nov 03 16:16:46.723379 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.723336    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgznh\" (UniqueName: \"kubernetes.io/projected/381eb05d-c83e-49b3-89b3-935ea62fb4e1-kube-api-access-rgznh\") pod \"payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c\" (UID: \"381eb05d-c83e-49b3-89b3-935ea62fb4e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"
Nov 03 16:16:46.724189 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.724161    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/381eb05d-c83e-49b3-89b3-935ea62fb4e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c\" (UID: \"381eb05d-c83e-49b3-89b3-935ea62fb4e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"
Nov 03 16:16:46.748499 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.748455    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgznh\" (UniqueName: \"kubernetes.io/projected/381eb05d-c83e-49b3-89b3-935ea62fb4e1-kube-api-access-rgznh\") pod \"payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c\" (UID: \"381eb05d-c83e-49b3-89b3-935ea62fb4e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"
Nov 03 16:16:46.795917 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.795882    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"]
Nov 03 16:16:46.802194 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.802170    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:46.812129 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.811941    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"
Nov 03 16:16:46.824277 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.818551    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-c09b17-serving-cert\""
Nov 03 16:16:46.824277 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.818950    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:16:46.824277 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.819039    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 16:16:46.824277 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.819288    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:16:46.824277 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.819897    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v65q5\""
Nov 03 16:16:46.824277 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.822122    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"]
Nov 03 16:16:46.926663 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.926622    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:46.927051 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.927030    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmmrz\" (UniqueName: \"kubernetes.io/projected/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-kube-api-access-gmmrz\") pod \"maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:46.927280 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.927262    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:46.927491 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.927475    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-metrics-certs\") pod \"maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:46.961761 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:46.961722    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:47.029095 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.029057    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:47.029594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.029170    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmmrz\" (UniqueName: \"kubernetes.io/projected/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-kube-api-access-gmmrz\") pod \"maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:47.029594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.029227    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:47.029594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.029266    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-metrics-certs\") pod \"maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:47.029862 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.029839    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:47.038870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.033588    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:47.038870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.035248    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-metrics-certs\") pod \"maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:47.039271 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.039013    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"]
Nov 03 16:16:47.046037 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.046017    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"
Nov 03 16:16:47.053952 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.053409    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmmrz\" (UniqueName: \"kubernetes.io/projected/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-kube-api-access-gmmrz\") pod \"maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:47.054901 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.054843    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"]
Nov 03 16:16:47.138268 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.138188    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:47.139537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.139507    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-4a19cd\""
Nov 03 16:16:47.139742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.139624    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-4a19cd-dockercfg-s94w9\""
Nov 03 16:16:47.175024 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.174982    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"]
Nov 03 16:16:47.177516 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.177426    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:47.240845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.239650    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39813549-9eb7-4cb7-9f08-4bf983b4cebc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz\" (UID: \"39813549-9eb7-4cb7-9f08-4bf983b4cebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"
Nov 03 16:16:47.240845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.239717    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4242c\" (UniqueName: \"kubernetes.io/projected/39813549-9eb7-4cb7-9f08-4bf983b4cebc-kube-api-access-4242c\") pod \"payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz\" (UID: \"39813549-9eb7-4cb7-9f08-4bf983b4cebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"
Nov 03 16:16:47.340702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.340625    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39813549-9eb7-4cb7-9f08-4bf983b4cebc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz\" (UID: \"39813549-9eb7-4cb7-9f08-4bf983b4cebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"
Nov 03 16:16:47.340702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.340689    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4242c\" (UniqueName: \"kubernetes.io/projected/39813549-9eb7-4cb7-9f08-4bf983b4cebc-kube-api-access-4242c\") pod \"payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz\" (UID: \"39813549-9eb7-4cb7-9f08-4bf983b4cebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"
Nov 03 16:16:47.341553 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.341527    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39813549-9eb7-4cb7-9f08-4bf983b4cebc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz\" (UID: \"39813549-9eb7-4cb7-9f08-4bf983b4cebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"
Nov 03 16:16:47.355197 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.355160    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4242c\" (UniqueName: \"kubernetes.io/projected/39813549-9eb7-4cb7-9f08-4bf983b4cebc-kube-api-access-4242c\") pod \"payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz\" (UID: \"39813549-9eb7-4cb7-9f08-4bf983b4cebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"
Nov 03 16:16:47.389561 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.389525    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"
Nov 03 16:16:47.549302 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.549226    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"]
Nov 03 16:16:47.764696 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.764594    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"]
Nov 03 16:16:47.770344 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:16:47.770305    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39813549_9eb7_4cb7_9f08_4bf983b4cebc.slice/crio-cb6ddd490e72c4bb96f7a3634a095526b434afbaea241b699406c6699ed38a3e WatchSource:0}: Error finding container cb6ddd490e72c4bb96f7a3634a095526b434afbaea241b699406c6699ed38a3e: Status 404 returned error can't find the container with id cb6ddd490e72c4bb96f7a3634a095526b434afbaea241b699406c6699ed38a3e
Nov 03 16:16:47.780660 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.780588    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c" event={"ID":"381eb05d-c83e-49b3-89b3-935ea62fb4e1","Type":"ContainerStarted","Data":"9f754514801aaf46eea452fcaa0fffd08c56f25ca371734a4aa6678dfda11816"}
Nov 03 16:16:47.782919 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:47.782835    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4" event={"ID":"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742","Type":"ContainerStarted","Data":"c97b5665e2129e890bd4af850a5a749aa832bb21d22f6de7a79d1ccb112cd2f9"}
Nov 03 16:16:48.165388 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:48.165275    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:48.387743 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:48.387631    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:48.568761 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:48.568729    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:48.795887 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:48.795178    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2" event={"ID":"f5d53d56-ee8a-4465-8a5a-06e9064ab695","Type":"ContainerStarted","Data":"c294d8fa8288047d98fbfa384ed565250c49758c4b9a7873ea128d30aeaa021b"}
Nov 03 16:16:48.799592 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:48.796485    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"
Nov 03 16:16:48.816817 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:48.806836    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz" event={"ID":"39813549-9eb7-4cb7-9f08-4bf983b4cebc","Type":"ContainerStarted","Data":"cb6ddd490e72c4bb96f7a3634a095526b434afbaea241b699406c6699ed38a3e"}
Nov 03 16:16:48.839047 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:48.826394    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2" podStartSLOduration=1.663848648 podStartE2EDuration="5.82637557s" totalImagesPullingTime="4.162526922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:48.824241913 +0000 UTC m=+822.837640061" watchObservedRunningTime="2026-11-03 16:16:48.82637557 +0000 UTC m=+822.839773717"
Nov 03 16:16:48.839047 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:48.832325    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t" event={"ID":"ccadaa1e-d59a-4231-ad4c-d8f46e7f4365","Type":"ContainerStarted","Data":"e1f0b88be24b3429927184dd1f7e14df0d93aa3ba1a9ee036d331725d9fa1cad"}
Nov 03 16:16:48.839047 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:48.833298    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"
Nov 03 16:16:48.855953 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:48.855890    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t" podStartSLOduration=1.548191834 podStartE2EDuration="5.855870235s" totalImagesPullingTime="4.307678401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:48.851994505 +0000 UTC m=+822.865392678" watchObservedRunningTime="2026-11-03 16:16:48.855870235 +0000 UTC m=+822.869268387"
Nov 03 16:16:49.154509 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.154467    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:49.246511 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.246479    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"]
Nov 03 16:16:49.254962 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.254129    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"
Nov 03 16:16:49.260382 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.260304    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"]
Nov 03 16:16:49.264480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.264362    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d569fb-dockercfg-ghdkc\""
Nov 03 16:16:49.264480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.264447    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d569fb\""
Nov 03 16:16:49.337142 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.336981    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:49.375606 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.375544    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mstc\" (UniqueName: \"kubernetes.io/projected/a78c2dad-d700-4697-9220-c08a07a3d4c3-kube-api-access-9mstc\") pod \"payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt\" (UID: \"a78c2dad-d700-4697-9220-c08a07a3d4c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"
Nov 03 16:16:49.375808 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.375627    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78c2dad-d700-4697-9220-c08a07a3d4c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt\" (UID: \"a78c2dad-d700-4697-9220-c08a07a3d4c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"
Nov 03 16:16:49.477992 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.477025    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mstc\" (UniqueName: \"kubernetes.io/projected/a78c2dad-d700-4697-9220-c08a07a3d4c3-kube-api-access-9mstc\") pod \"payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt\" (UID: \"a78c2dad-d700-4697-9220-c08a07a3d4c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"
Nov 03 16:16:49.477992 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.477073    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78c2dad-d700-4697-9220-c08a07a3d4c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt\" (UID: \"a78c2dad-d700-4697-9220-c08a07a3d4c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"
Nov 03 16:16:49.477992 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.477674    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78c2dad-d700-4697-9220-c08a07a3d4c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt\" (UID: \"a78c2dad-d700-4697-9220-c08a07a3d4c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"
Nov 03 16:16:49.497247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.497201    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mstc\" (UniqueName: \"kubernetes.io/projected/a78c2dad-d700-4697-9220-c08a07a3d4c3-kube-api-access-9mstc\") pod \"payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt\" (UID: \"a78c2dad-d700-4697-9220-c08a07a3d4c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"
Nov 03 16:16:49.577838 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.577784    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"
Nov 03 16:16:49.850387 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.850302    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz" event={"ID":"39813549-9eb7-4cb7-9f08-4bf983b4cebc","Type":"ContainerStarted","Data":"119250b5231ab824d8f19f7131df5e0ceb47586ebb4089e30acd9c790c9f6a29"}
Nov 03 16:16:49.850545 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.850429    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"
Nov 03 16:16:49.877669 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.877609    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz" podStartSLOduration=1.647117894 podStartE2EDuration="2.877589893s" totalImagesPullingTime="1.230471999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:49.875177719 +0000 UTC m=+823.888575865" watchObservedRunningTime="2026-11-03 16:16:49.877589893 +0000 UTC m=+823.890988038"
Nov 03 16:16:49.984945 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:49.984870    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"]
Nov 03 16:16:49.989126 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:16:49.988015    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78c2dad_d700_4697_9220_c08a07a3d4c3.slice/crio-521c9997d9758b05db6aaec0ffac0750b7e1da4518a04268d8654df9d15638b8 WatchSource:0}: Error finding container 521c9997d9758b05db6aaec0ffac0750b7e1da4518a04268d8654df9d15638b8: Status 404 returned error can't find the container with id 521c9997d9758b05db6aaec0ffac0750b7e1da4518a04268d8654df9d15638b8
Nov 03 16:16:50.076275 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:50.076223    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:50.337137 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:50.337098    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:50.857261 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:50.857225    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c" event={"ID":"381eb05d-c83e-49b3-89b3-935ea62fb4e1","Type":"ContainerStarted","Data":"c86b1a39aff6e45af98ab2633d29df483eec7f5621650ed9df88268e0f8d7efd"}
Nov 03 16:16:50.857500 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:50.857402    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"
Nov 03 16:16:50.859999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:50.859956    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt" event={"ID":"a78c2dad-d700-4697-9220-c08a07a3d4c3","Type":"ContainerStarted","Data":"aeeff5f3f449f200493af9d617a18fe75e0fbb61fb59a2fb8eba8ee68de0a604"}
Nov 03 16:16:50.859999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:50.860024    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt" event={"ID":"a78c2dad-d700-4697-9220-c08a07a3d4c3","Type":"ContainerStarted","Data":"521c9997d9758b05db6aaec0ffac0750b7e1da4518a04268d8654df9d15638b8"}
Nov 03 16:16:50.860993 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:50.860947    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"
Nov 03 16:16:50.863634 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:50.863600    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n" event={"ID":"dde0ec03-70f8-47bb-af69-6a56524d425f","Type":"ContainerStarted","Data":"c2ea69746fae98fd8d8694fa50e08db9670abda8c0717fd6beb5e3588399bb33"}
Nov 03 16:16:50.864428 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:50.864360    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"
Nov 03 16:16:50.866493 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:50.866433    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g" event={"ID":"80ee90cc-3ecf-40be-9d34-511fb0b42721","Type":"ContainerStarted","Data":"feebba58ac9d4615c0bb7af2125418be4b4f25dd8aa62b7a925a437c7d372fc2"}
Nov 03 16:16:50.866711 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:50.866670    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"
Nov 03 16:16:50.880904 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:50.880844    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c" podStartSLOduration=1.680998111 podStartE2EDuration="4.880829849s" totalImagesPullingTime="3.199831738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:50.877509203 +0000 UTC m=+824.890907375" watchObservedRunningTime="2026-11-03 16:16:50.880829849 +0000 UTC m=+824.894227997"
Nov 03 16:16:50.934720 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:50.934659    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt" podStartSLOduration=1.934639614 podStartE2EDuration="1.934639614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:50.90391276 +0000 UTC m=+824.917310918" watchObservedRunningTime="2026-11-03 16:16:50.934639614 +0000 UTC m=+824.948037762"
Nov 03 16:16:50.955880 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:50.955831    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n" podStartSLOduration=1.720012221 podStartE2EDuration="6.955817035s" totalImagesPullingTime="5.235804814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:50.95402862 +0000 UTC m=+824.967426768" watchObservedRunningTime="2026-11-03 16:16:50.955817035 +0000 UTC m=+824.969215175"
Nov 03 16:16:50.985293 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:50.985239    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g" podStartSLOduration=2.3243760030000002 podStartE2EDuration="6.98522408s" totalImagesPullingTime="4.660848077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:50.983455691 +0000 UTC m=+824.996853838" watchObservedRunningTime="2026-11-03 16:16:50.98522408 +0000 UTC m=+824.998622225"
Nov 03 16:16:51.235516 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:51.235413    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:51.545819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:51.536445    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:52.348672 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:52.348637    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:52.542610 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:52.542563    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:53.343923 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.343870    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"]
Nov 03 16:16:53.372877 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.371700    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"]
Nov 03 16:16:53.372877 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.371854    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:16:53.377144 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.377115    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-afb990-serving-cert\""
Nov 03 16:16:53.464420 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.464392    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:53.498000 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.497963    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab2854b-0c08-4417-a5cf-bf67c191b51f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:16:53.498173 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.498008    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab2854b-0c08-4417-a5cf-bf67c191b51f-metrics-certs\") pod \"maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:16:53.498173 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.498038    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfh7\" (UniqueName: \"kubernetes.io/projected/bab2854b-0c08-4417-a5cf-bf67c191b51f-kube-api-access-mtfh7\") pod \"maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:16:53.498173 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.498140    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab2854b-0c08-4417-a5cf-bf67c191b51f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:16:53.599444 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.599411    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab2854b-0c08-4417-a5cf-bf67c191b51f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:16:53.599444 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.599451    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab2854b-0c08-4417-a5cf-bf67c191b51f-metrics-certs\") pod \"maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:16:53.599656 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.599470    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfh7\" (UniqueName: \"kubernetes.io/projected/bab2854b-0c08-4417-a5cf-bf67c191b51f-kube-api-access-mtfh7\") pod \"maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:16:53.599656 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.599502    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab2854b-0c08-4417-a5cf-bf67c191b51f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:16:53.600219 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.600191    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab2854b-0c08-4417-a5cf-bf67c191b51f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:16:53.602272 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.602250    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab2854b-0c08-4417-a5cf-bf67c191b51f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:16:53.602376 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.602365    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab2854b-0c08-4417-a5cf-bf67c191b51f-metrics-certs\") pod \"maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:16:53.622097 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.622067    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfh7\" (UniqueName: \"kubernetes.io/projected/bab2854b-0c08-4417-a5cf-bf67c191b51f-kube-api-access-mtfh7\") pod \"maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:16:53.666574 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.666445    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:53.688643 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.688615    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:16:53.891612 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.891572    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4" event={"ID":"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742","Type":"ContainerStarted","Data":"9464c131233cd3d78eeaee9ff7c0beeb74c228ada948e87538e0691f3cafa9a1"}
Nov 03 16:16:53.891755 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.891746    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:53.960934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:53.916115    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4" podStartSLOduration=2.370333407 podStartE2EDuration="7.916095173s" totalImagesPullingTime="5.545761766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:53.915211572 +0000 UTC m=+827.928609720" watchObservedRunningTime="2026-11-03 16:16:53.916095173 +0000 UTC m=+827.929493319"
Nov 03 16:16:54.040982 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:54.040957    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"]
Nov 03 16:16:54.043713 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:16:54.043680    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab2854b_0c08_4417_a5cf_bf67c191b51f.slice/crio-72ec86a0c5b8a5e3cca24c46cf089dd5b4ba1a26d3a4ea456323795d69c0eff6 WatchSource:0}: Error finding container 72ec86a0c5b8a5e3cca24c46cf089dd5b4ba1a26d3a4ea456323795d69c0eff6: Status 404 returned error can't find the container with id 72ec86a0c5b8a5e3cca24c46cf089dd5b4ba1a26d3a4ea456323795d69c0eff6
Nov 03 16:16:54.487987 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:54.487951    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:54.898584 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:54.898549    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw" event={"ID":"bab2854b-0c08-4417-a5cf-bf67c191b51f","Type":"ContainerStarted","Data":"ccb5141d3367ac6dd7970f0f7be77e53af74b3dcc9416bc82779debfc54e6c6c"}
Nov 03 16:16:54.898832 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:54.898783    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:16:54.898909 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:54.898861    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw" event={"ID":"bab2854b-0c08-4417-a5cf-bf67c191b51f","Type":"ContainerStarted","Data":"72ec86a0c5b8a5e3cca24c46cf089dd5b4ba1a26d3a4ea456323795d69c0eff6"}
Nov 03 16:16:54.933705 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:54.933254    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw" podStartSLOduration=1.450344745 podStartE2EDuration="1.933232894s" totalImagesPullingTime="482.888149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:54.926190835 +0000 UTC m=+828.939588981" watchObservedRunningTime="2026-11-03 16:16:54.933232894 +0000 UTC m=+828.946631044"
Nov 03 16:16:55.343562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:55.343520    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:56.278698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:56.278663    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:57.469177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:57.469134    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:57.755010 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:57.754918    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:16:57.755010 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:57.754971    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt" podUID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:16:58.798672 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:16:58.798619    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:16:58.954629 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:58.954588    6506 generic.go:356] "Generic (PLEG): container finished" podID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerID="cd4df6ddd25a97f54b079e7943d6c6c6791ad373e3e2727e1f3550a0f0c01aab" exitCode=0
Nov 03 16:16:58.954998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:58.954913    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt" event={"ID":"15f2a444-a2bf-46ed-9a2d-4a40dd09c537","Type":"ContainerDied","Data":"cd4df6ddd25a97f54b079e7943d6c6c6791ad373e3e2727e1f3550a0f0c01aab"}
Nov 03 16:16:58.955618 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:58.955596    6506 scope.go:122] "RemoveContainer" containerID="cd4df6ddd25a97f54b079e7943d6c6c6791ad373e3e2727e1f3550a0f0c01aab"
Nov 03 16:16:58.957506 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:58.957471    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:16:58.957617 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:58.957543    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:16:58.957617 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:58.957574    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:16:59.176535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:59.176496    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:16:59.905063 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:59.905032    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:16:59.985938 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:16:59.984834    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt" event={"ID":"15f2a444-a2bf-46ed-9a2d-4a40dd09c537","Type":"ContainerStarted","Data":"111d7eb51a6e97160d672581c5ec67686cf513d636ed2e4e9ad6c18aef11e9e1"}
Nov 03 16:17:00.561694 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:00.561352    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:00.769607 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:00.769566    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:17:00.769860 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:00.769837    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 16:17:00.769962 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:00.769889    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt" podUID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 16:17:00.870216 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:00.870130    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"
Nov 03 16:17:00.870216 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:00.870197    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"
Nov 03 16:17:00.870463 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:00.870226    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"
Nov 03 16:17:00.887357 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:00.887327    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:00.965973 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:00.965939    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:17:01.060863 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:01.057309    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:01.769697 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:01.769233    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 16:17:01.769697 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:01.769297    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt" podUID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 16:17:01.877536 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:01.877507    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"
Nov 03 16:17:01.877708 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:01.877575    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"
Nov 03 16:17:02.290397 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:02.290367    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:02.457399 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:02.457370    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:02.628010 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:02.627846    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:02.769691 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:02.769656    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 16:17:02.769858 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:02.769727    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt" podUID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 16:17:02.887891 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:02.887782    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"
Nov 03 16:17:02.888218 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:02.888199    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"
Nov 03 16:17:03.001324 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:02.996823    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:03.044470 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:03.044420    6506 generic.go:356] "Generic (PLEG): container finished" podID="e0ed082c-356f-45b5-80e5-c70a6f5d055d" containerID="36ae5eab43b7543917c52fcfb1337e71d5e7c819767dac31f60642adb1f4e97b" exitCode=0
Nov 03 16:17:03.044470 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:03.044462    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5" event={"ID":"e0ed082c-356f-45b5-80e5-c70a6f5d055d","Type":"ContainerDied","Data":"36ae5eab43b7543917c52fcfb1337e71d5e7c819767dac31f60642adb1f4e97b"}
Nov 03 16:17:03.044974 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:03.044954    6506 scope.go:122] "RemoveContainer" containerID="36ae5eab43b7543917c52fcfb1337e71d5e7c819767dac31f60642adb1f4e97b"
Nov 03 16:17:03.046425 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:03.046399    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:17:03.046514 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:03.046461    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:17:03.046580 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:03.046539    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:17:03.204023 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:03.203989    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:03.438843 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:03.438716    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:03.613821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:03.613192    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:03.771840 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:03.769957    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 16:17:03.771840 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:03.770015    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt" podUID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 16:17:03.804485 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:03.804447    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:04.064268 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:04.064175    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5" event={"ID":"e0ed082c-356f-45b5-80e5-c70a6f5d055d","Type":"ContainerStarted","Data":"f2136fcf41133c45299562b0c9bf9e4b86d61c40cb9f52092b95fd0a5ac329ac"}
Nov 03 16:17:04.248530 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:04.248487    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:04.547821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:04.540272    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:04.768921 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:04.768877    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 16:17:04.769115 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:04.768940    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt" podUID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 16:17:04.837868 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:04.836894    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:04.901625 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:04.901592    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:17:04.901816 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:04.901705    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 16:17:04.901816 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:04.901740    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5" podUID="e0ed082c-356f-45b5-80e5-c70a6f5d055d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 16:17:05.248064 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:05.247873    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:05.773147 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:05.773118    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:17:05.773642 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:05.773604    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:17:05.774346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:05.774330    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:17:05.901039 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:05.900998    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 16:17:05.901195 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:05.901057    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5" podUID="e0ed082c-356f-45b5-80e5-c70a6f5d055d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 16:17:06.331327 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:06.331256    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:06.900972 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:06.900939    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 16:17:06.901436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:06.901006    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5" podUID="e0ed082c-356f-45b5-80e5-c70a6f5d055d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 16:17:07.282543 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:07.282504    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:07.681922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:07.681755    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:07.904818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:07.904460    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 16:17:07.904818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:07.904546    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5" podUID="e0ed082c-356f-45b5-80e5-c70a6f5d055d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 16:17:08.094737 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:08.094700    6506 generic.go:356] "Generic (PLEG): container finished" podID="20b9b08d-2d97-4604-890c-10f6305c0108" containerID="1551a6edd5f3fcca5c624548874ffe5096f2c67fe1cd90d38c89ec5e8d7f553a" exitCode=0
Nov 03 16:17:08.095037 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:08.094872    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw" event={"ID":"20b9b08d-2d97-4604-890c-10f6305c0108","Type":"ContainerDied","Data":"1551a6edd5f3fcca5c624548874ffe5096f2c67fe1cd90d38c89ec5e8d7f553a"}
Nov 03 16:17:08.095644 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:08.095623    6506 scope.go:122] "RemoveContainer" containerID="1551a6edd5f3fcca5c624548874ffe5096f2c67fe1cd90d38c89ec5e8d7f553a"
Nov 03 16:17:08.100487 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:08.100449    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:17:08.100974 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:08.100539    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:17:08.100974 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:08.100593    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:17:08.337347 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:08.337311    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:08.537826 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:08.534298    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:08.901378 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:08.901299    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 16:17:08.901378 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:08.901362    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5" podUID="e0ed082c-356f-45b5-80e5-c70a6f5d055d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 16:17:09.070844 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:09.070788    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:09.102096 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:09.102068    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw" event={"ID":"20b9b08d-2d97-4604-890c-10f6305c0108","Type":"ContainerStarted","Data":"7234d9df4269a85af5e0ed9fe525aaa89ea6ed11dbe23c25ddeec25105b25298"}
Nov 03 16:17:09.271211 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:09.271178    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:09.379892 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:09.379847    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:17:09.381300 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:09.381268    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 16:17:09.381450 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:09.381319    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw" podUID="20b9b08d-2d97-4604-890c-10f6305c0108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 16:17:09.904843 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:09.904814    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:17:09.905090 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:09.905067    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:17:09.905901 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:09.905881    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:17:10.132977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:10.132926    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:10.297318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:10.297281    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:10.380076 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:10.380041    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 16:17:10.380243 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:10.380097    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw" podUID="20b9b08d-2d97-4604-890c-10f6305c0108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 16:17:11.324821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:11.323220    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:11.379806 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:11.379758    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 16:17:11.379966 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:11.379832    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw" podUID="20b9b08d-2d97-4604-890c-10f6305c0108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 16:17:11.731933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:11.728905    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:17:12.380050 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:12.380012    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 16:17:12.380488 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:12.380080    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw" podUID="20b9b08d-2d97-4604-890c-10f6305c0108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 16:17:12.739420 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:12.739336    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:13.380562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:13.380532    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 16:17:13.381004 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:13.380591    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw" podUID="20b9b08d-2d97-4604-890c-10f6305c0108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 16:17:14.318647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.318604    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"]
Nov 03 16:17:14.322506 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.322491    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.341970 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.341941    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d569fb\"/\"openshift-service-ca.crt\""
Nov 03 16:17:14.342110 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.341994    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d569fb\"/\"kube-root-ca.crt\""
Nov 03 16:17:14.342859 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.342836    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"]
Nov 03 16:17:14.344844 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.344768    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d569fb\"/\"default-dockercfg-dq8hv\""
Nov 03 16:17:14.346037 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.345982    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d569fb\"/\"simulator-w0-d569fb-kserve-self-signed-certs\""
Nov 03 16:17:14.360619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.360593    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kckvk\" (UniqueName: \"kubernetes.io/projected/ad1e4d9f-21b8-40a7-a919-c53ce248510e-kube-api-access-kckvk\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.360866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.360842    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad1e4d9f-21b8-40a7-a919-c53ce248510e-tls-certs\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.360988 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.360912    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-tmp-dir\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.361077 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.361027    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-dshm\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.361147 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.361094    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-home\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.361227 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.361208    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-model-cache\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.462035 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.461998    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-tmp-dir\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.462534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.462053    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-dshm\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.462534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.462091    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-home\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.462534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.462145    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-model-cache\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.462534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.462185    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kckvk\" (UniqueName: \"kubernetes.io/projected/ad1e4d9f-21b8-40a7-a919-c53ce248510e-kube-api-access-kckvk\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.462534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.462243    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad1e4d9f-21b8-40a7-a919-c53ce248510e-tls-certs\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.462534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.462455    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-tmp-dir\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.462878 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.462843    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-model-cache\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.462979 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.462962    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-home\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.465054 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.465029    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:17:14.465554 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.465529    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:14.465852 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.465706    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad1e4d9f-21b8-40a7-a919-c53ce248510e-tls-certs\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.466657 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.466630    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-dshm\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.467130 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.467111    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:17:14.475273 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.475247    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kckvk\" (UniqueName: \"kubernetes.io/projected/ad1e4d9f-21b8-40a7-a919-c53ce248510e-kube-api-access-kckvk\") pod \"simulator-w0-d569fb-kserve-587868d9d9-mflh5\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") " pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.635347 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.635257    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:14.921655 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:14.921571    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"]
Nov 03 16:17:14.923998 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:17:14.923958    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad1e4d9f_21b8_40a7_a919_c53ce248510e.slice/crio-33dd168bea80448824018aacbc3274eb1a2feb0500189d3568163dfdd4b50908 WatchSource:0}: Error finding container 33dd168bea80448824018aacbc3274eb1a2feb0500189d3568163dfdd4b50908: Status 404 returned error can't find the container with id 33dd168bea80448824018aacbc3274eb1a2feb0500189d3568163dfdd4b50908
Nov 03 16:17:15.159954 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:15.159920    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" event={"ID":"ad1e4d9f-21b8-40a7-a919-c53ce248510e","Type":"ContainerStarted","Data":"01663f319542374ffb8c089a265a598191d28da35f82cc345fdc424186562336"}
Nov 03 16:17:15.160120 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:15.159966    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" event={"ID":"ad1e4d9f-21b8-40a7-a919-c53ce248510e","Type":"ContainerStarted","Data":"33dd168bea80448824018aacbc3274eb1a2feb0500189d3568163dfdd4b50908"}
Nov 03 16:17:15.180592 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:15.180510    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" podStartSLOduration=1.180497142 podStartE2EDuration="1.180497142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:15.17964819 +0000 UTC m=+849.193046337" watchObservedRunningTime="2026-11-03 16:17:15.180497142 +0000 UTC m=+849.193895288"
Nov 03 16:17:15.635485 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:15.635454    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:15.772099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:15.772058    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:17.375596 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:17.375566    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:18.523145 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:18.523113    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:19.384739 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:19.384689    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:20.126202 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:20.126165    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:20.372078 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:20.372051    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:21.402460 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:21.402430    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:21.892456 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:21.892423    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:22.604474 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:22.604439    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:23.363325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.363288    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:23.470713 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.470682    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"]
Nov 03 16:17:23.474448 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.474427    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.479250 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.479229    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-afb990\"/\"simulator-w6-afb990-kserve-self-signed-certs\""
Nov 03 16:17:23.484015 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.482105    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-afb990\"/\"default-dockercfg-z6l22\""
Nov 03 16:17:23.484015 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.482828    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-afb990\"/\"openshift-service-ca.crt\""
Nov 03 16:17:23.484015 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.483424    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-afb990\"/\"kube-root-ca.crt\""
Nov 03 16:17:23.486386 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.486368    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"]
Nov 03 16:17:23.564149 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.564108    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:23.579510 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.579483    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-model-cache\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.579664 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.579522    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-dshm\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.579664 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.579548    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-home\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.579757 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.579696    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-tmp-dir\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.579814 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.579761    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf9062b-2e57-4875-a717-a89d753afa28-tls-certs\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.579896 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.579877    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9fzz\" (UniqueName: \"kubernetes.io/projected/7cf9062b-2e57-4875-a717-a89d753afa28-kube-api-access-h9fzz\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.681126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.681025    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-tmp-dir\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.681695 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.681667    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-tmp-dir\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.681817 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.681681    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf9062b-2e57-4875-a717-a89d753afa28-tls-certs\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.681982 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.681963    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fzz\" (UniqueName: \"kubernetes.io/projected/7cf9062b-2e57-4875-a717-a89d753afa28-kube-api-access-h9fzz\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.682132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.682118    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-model-cache\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.682252 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.682239    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-dshm\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.682362 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.682349    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-home\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.682841 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.682820    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-home\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.683230 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.683210    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-model-cache\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.688121 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.688100    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-dshm\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.692429 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.692405    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf9062b-2e57-4875-a717-a89d753afa28-tls-certs\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.695268 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.695242    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9fzz\" (UniqueName: \"kubernetes.io/projected/7cf9062b-2e57-4875-a717-a89d753afa28-kube-api-access-h9fzz\") pod \"simulator-w6-afb990-kserve-5f658f4968-bl5jf\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") " pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:23.787942 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:23.787906    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:24.071943 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:24.071918    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"]
Nov 03 16:17:24.073247 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:17:24.073219    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cf9062b_2e57_4875_a717_a89d753afa28.slice/crio-e979ef782f11a5711c07f23307dcf90f239264e71a8548b7b0683233491b8018 WatchSource:0}: Error finding container e979ef782f11a5711c07f23307dcf90f239264e71a8548b7b0683233491b8018: Status 404 returned error can't find the container with id e979ef782f11a5711c07f23307dcf90f239264e71a8548b7b0683233491b8018
Nov 03 16:17:24.220995 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:24.220959    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" event={"ID":"7cf9062b-2e57-4875-a717-a89d753afa28","Type":"ContainerStarted","Data":"b23f829d209c30f065c601132de685b661f89401a15cf2008feac8ded822f575"}
Nov 03 16:17:24.221190 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:24.221004    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" event={"ID":"7cf9062b-2e57-4875-a717-a89d753afa28","Type":"ContainerStarted","Data":"e979ef782f11a5711c07f23307dcf90f239264e71a8548b7b0683233491b8018"}
Nov 03 16:17:24.242987 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:24.242931    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podStartSLOduration=1.242919182 podStartE2EDuration="1.242919182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:24.240762895 +0000 UTC m=+858.254161043" watchObservedRunningTime="2026-11-03 16:17:24.242919182 +0000 UTC m=+858.256317328"
Nov 03 16:17:24.286698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:24.286663    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:24.635422 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:24.635383    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:24.647574 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:24.647549    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:24.788979 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:24.788953    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:25.013562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:25.013448    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:25.239534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:25.239506    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:17:25.822191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:25.822155    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:26.524773 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:26.524736    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:27.354048 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:27.354011    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:27.755783 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:27.755701    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:28.299961 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:28.299926    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:28.796165 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:17:28.796131    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:17:29.027572 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:29.027529    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:29.812599 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:29.812562    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:30.582340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.582304    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"]
Nov 03 16:17:30.586004 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.585984    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.601430 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.601406    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b132e3\"/\"premium-w2-b132e3-kserve-self-signed-certs\""
Nov 03 16:17:30.603518 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.603488    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b132e3\"/\"default-dockercfg-55z92\""
Nov 03 16:17:30.605590 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.605558    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"]
Nov 03 16:17:30.607787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.607770    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b132e3\"/\"openshift-service-ca.crt\""
Nov 03 16:17:30.607895 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.607834    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b132e3\"/\"kube-root-ca.crt\""
Nov 03 16:17:30.662460 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.662423    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:30.668472 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.668451    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-home\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.668553 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.668490    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-tmp-dir\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.668601 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.668590    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fb2203-adc1-4dcb-b092-42f6d5296165-tls-certs\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.668657 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.668645    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-model-cache\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.668731 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.668712    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmmh6\" (UniqueName: \"kubernetes.io/projected/c7fb2203-adc1-4dcb-b092-42f6d5296165-kube-api-access-dmmh6\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.668766 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.668749    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-dshm\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.770087 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.770047    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-model-cache\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.770266 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.770091    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmmh6\" (UniqueName: \"kubernetes.io/projected/c7fb2203-adc1-4dcb-b092-42f6d5296165-kube-api-access-dmmh6\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.770266 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.770126    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-dshm\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.770266 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.770162    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-home\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.770266 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.770197    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-tmp-dir\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.770445 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.770285    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fb2203-adc1-4dcb-b092-42f6d5296165-tls-certs\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.770543 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.770522    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-model-cache\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.770607 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.770574    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-home\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.770669 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.770608    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-tmp-dir\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.772540 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.772521    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-dshm\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.772913 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.772892    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fb2203-adc1-4dcb-b092-42f6d5296165-tls-certs\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.783385 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.783366    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmmh6\" (UniqueName: \"kubernetes.io/projected/c7fb2203-adc1-4dcb-b092-42f6d5296165-kube-api-access-dmmh6\") pod \"premium-w2-b132e3-kserve-7c7455fb84-zd88g\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") " pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:30.843759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.843677    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:30.898451 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:30.898421    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:31.185651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:31.185621    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"]
Nov 03 16:17:31.187705 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:17:31.187678    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7fb2203_adc1_4dcb_b092_42f6d5296165.slice/crio-06bb32be6465a62202a4806a079f21d46884004870468762c492832a4b01379e WatchSource:0}: Error finding container 06bb32be6465a62202a4806a079f21d46884004870468762c492832a4b01379e: Status 404 returned error can't find the container with id 06bb32be6465a62202a4806a079f21d46884004870468762c492832a4b01379e
Nov 03 16:17:31.263096 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:31.263063    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" event={"ID":"c7fb2203-adc1-4dcb-b092-42f6d5296165","Type":"ContainerStarted","Data":"8ccdd1537bfd5d5df8316ed0a27f604e97e1abecad3ea3284341e8e551a778e2"}
Nov 03 16:17:31.263236 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:31.263111    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" event={"ID":"c7fb2203-adc1-4dcb-b092-42f6d5296165","Type":"ContainerStarted","Data":"06bb32be6465a62202a4806a079f21d46884004870468762c492832a4b01379e"}
Nov 03 16:17:31.309627 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:31.309573    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podStartSLOduration=1.309555184 podStartE2EDuration="1.309555184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:31.300891694 +0000 UTC m=+865.314289840" watchObservedRunningTime="2026-11-03 16:17:31.309555184 +0000 UTC m=+865.322953333"
Nov 03 16:17:31.771788 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:31.771687    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:31.899519 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:31.899490    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:32.854655 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:32.854624    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:33.670284 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:33.670252    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:33.789093 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:33.789054    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:33.801512 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:33.801490    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:34.292229 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:34.292201    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:17:34.420357 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:34.420320    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:35.239881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:35.239846    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:36.022355 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:36.022324    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:36.873590 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:36.873551    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:37.620419 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:37.620383    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:37.798728 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:37.798696    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:38.538528 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:38.538493    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:39.306394 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.306354    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"]
Nov 03 16:17:39.313879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.312777    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.320261 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.320235    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-69cedd\"/\"default-dockercfg-vkbzw\""
Nov 03 16:17:39.321186 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.321167    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-69cedd\"/\"premium-w5-69cedd-kserve-self-signed-certs\""
Nov 03 16:17:39.322729 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.322472    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-69cedd\"/\"openshift-service-ca.crt\""
Nov 03 16:17:39.326466 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.326445    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"]
Nov 03 16:17:39.326578 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.326452    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-69cedd\"/\"kube-root-ca.crt\""
Nov 03 16:17:39.367325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.367295    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:39.368741 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.368715    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-model-cache\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.368932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.368763    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-dshm\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.368932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.368821    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-home\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.369035 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.368945    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-tmp-dir\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.369035 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.368980    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsrfc\" (UniqueName: \"kubernetes.io/projected/f4867d2d-c38c-44d6-924c-b0afe704f841-kube-api-access-bsrfc\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.369035 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.369014    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4867d2d-c38c-44d6-924c-b0afe704f841-tls-certs\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.470624 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.470589    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4867d2d-c38c-44d6-924c-b0afe704f841-tls-certs\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.471140 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.471118    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-model-cache\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.471447 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.471427    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-dshm\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.471712 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.471686    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-home\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.472124 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.472105    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-tmp-dir\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.472371 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.472348    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-tmp-dir\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.472507 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.471596    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-model-cache\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.472600 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.472365    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrfc\" (UniqueName: \"kubernetes.io/projected/f4867d2d-c38c-44d6-924c-b0afe704f841-kube-api-access-bsrfc\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.472735 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.472122    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-home\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.475577 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.475555    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4867d2d-c38c-44d6-924c-b0afe704f841-tls-certs\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.482930 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.482779    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-dshm\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.487000 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.486604    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsrfc\" (UniqueName: \"kubernetes.io/projected/f4867d2d-c38c-44d6-924c-b0afe704f841-kube-api-access-bsrfc\") pod \"premium-w5-69cedd-kserve-68977c5d54-cv58n\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") " pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.529724 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.529689    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:39.632156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.632076    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:39.929578 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:39.929500    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"]
Nov 03 16:17:39.934892 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:17:39.934862    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4867d2d_c38c_44d6_924c_b0afe704f841.slice/crio-62b15bb76137d23a27f0af9cda51125df6641e93ee2c6ca1552c831c44113e92 WatchSource:0}: Error finding container 62b15bb76137d23a27f0af9cda51125df6641e93ee2c6ca1552c831c44113e92: Status 404 returned error can't find the container with id 62b15bb76137d23a27f0af9cda51125df6641e93ee2c6ca1552c831c44113e92
Nov 03 16:17:40.326374 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:40.326339    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:40.327877 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:40.327839    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" event={"ID":"f4867d2d-c38c-44d6-924c-b0afe704f841","Type":"ContainerStarted","Data":"3c37186f351e58ff9984590f1350e79ad87fa9210b852e2cca3300ead48ee58d"}
Nov 03 16:17:40.328003 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:40.327885    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" event={"ID":"f4867d2d-c38c-44d6-924c-b0afe704f841","Type":"ContainerStarted","Data":"62b15bb76137d23a27f0af9cda51125df6641e93ee2c6ca1552c831c44113e92"}
Nov 03 16:17:40.350070 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:40.350027    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podStartSLOduration=1.350014308 podStartE2EDuration="1.350014308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:40.349040926 +0000 UTC m=+874.362439073" watchObservedRunningTime="2026-11-03 16:17:40.350014308 +0000 UTC m=+874.363412451"
Nov 03 16:17:40.632888 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:40.632784    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:40.899008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:40.898922    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:40.911552 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:40.911527    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:41.344365 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:41.344336    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:41.345487 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:41.345459    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:17:42.188033 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:42.187995    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:42.998254 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:42.998219    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:43.610680 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:43.610643    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:43.865243 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:43.865162    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:44.717937 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:44.717899    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:45.683491 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:45.683450    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:46.908681 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:46.908650    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:47.821091 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:47.821050    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:48.231863 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:48.231753    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:48.707551 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:48.707509    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:49.507300 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:49.507268    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:49.633200 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:49.633164    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:49.645727 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:49.645701    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:50.212067 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:50.212028    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:50.408618 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:50.408587    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:17:50.992739 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:50.992707    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:51.056902 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.056872    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"]
Nov 03 16:17:51.060562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.060546    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.069303 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.069271    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4a19cd\"/\"kube-root-ca.crt\""
Nov 03 16:17:51.071845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.071817    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"]
Nov 03 16:17:51.073247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.073230    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4a19cd\"/\"openshift-service-ca.crt\""
Nov 03 16:17:51.078192 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.078173    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4a19cd\"/\"premium-w4-4a19cd-kserve-self-signed-certs\""
Nov 03 16:17:51.086762 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.086732    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4a19cd\"/\"default-dockercfg-4mpx9\""
Nov 03 16:17:51.103083 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.103059    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-tmp-dir\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.103192 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.103110    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-home\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.103250 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.103184    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd55dad-fd00-47aa-84a0-3c38d7be189f-tls-certs\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.103293 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.103266    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-model-cache\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.103331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.103316    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8p6k\" (UniqueName: \"kubernetes.io/projected/3bd55dad-fd00-47aa-84a0-3c38d7be189f-kube-api-access-v8p6k\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.103379 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.103346    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-dshm\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.183433 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.183398    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:51.205115 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.205058    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-model-cache\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.205320 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.205166    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8p6k\" (UniqueName: \"kubernetes.io/projected/3bd55dad-fd00-47aa-84a0-3c38d7be189f-kube-api-access-v8p6k\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.205320 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.205202    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-dshm\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.205320 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.205292    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-tmp-dir\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.205476 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.205341    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-home\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.205476 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.205378    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd55dad-fd00-47aa-84a0-3c38d7be189f-tls-certs\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.205583 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.205524    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-model-cache\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.205844 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.205817    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-tmp-dir\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.206005 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.205844    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-home\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.208479 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.208448    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-dshm\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.208746 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.208729    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd55dad-fd00-47aa-84a0-3c38d7be189f-tls-certs\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.218629 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.218598    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8p6k\" (UniqueName: \"kubernetes.io/projected/3bd55dad-fd00-47aa-84a0-3c38d7be189f-kube-api-access-v8p6k\") pod \"premium-w4-4a19cd-kserve-9f6896644-7b6vf\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") " pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.374812 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.374770    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:51.657370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.657344    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"]
Nov 03 16:17:51.658929 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:17:51.658901    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bd55dad_fd00_47aa_84a0_3c38d7be189f.slice/crio-0acc0ed53e4a84b0cbdbb6f3e622d5799f20bf7a77a27bc397b223c103c516e0 WatchSource:0}: Error finding container 0acc0ed53e4a84b0cbdbb6f3e622d5799f20bf7a77a27bc397b223c103c516e0: Status 404 returned error can't find the container with id 0acc0ed53e4a84b0cbdbb6f3e622d5799f20bf7a77a27bc397b223c103c516e0
Nov 03 16:17:51.834738 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:51.834698    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:52.414221 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:52.414179    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" event={"ID":"3bd55dad-fd00-47aa-84a0-3c38d7be189f","Type":"ContainerStarted","Data":"0d05bfa15953053ae1a9eff19166c28a92848cc2c0092f99a618ccf142f15b5d"}
Nov 03 16:17:52.414713 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:52.414228    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" event={"ID":"3bd55dad-fd00-47aa-84a0-3c38d7be189f","Type":"ContainerStarted","Data":"0acc0ed53e4a84b0cbdbb6f3e622d5799f20bf7a77a27bc397b223c103c516e0"}
Nov 03 16:17:52.439582 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:52.439520    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podStartSLOduration=1.439499583 podStartE2EDuration="1.439499583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:52.43870625 +0000 UTC m=+886.452104397" watchObservedRunningTime="2026-11-03 16:17:52.439499583 +0000 UTC m=+886.452897730"
Nov 03 16:17:53.035650 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:53.035617    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:53.375532 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:53.375439    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:17:54.105339 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:54.105295    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:54.909536 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:54.909499    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:55.619172 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:55.619137    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:56.406628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:56.406579    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:56.653309 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:56.653260    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:57.295232 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:57.295193    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:58.116552 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:58.116514    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:58.798274 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:17:58.798234    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:17:58.912437 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:58.912369    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:59.487126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.487095    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"]
Nov 03 16:17:59.492375 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.492354    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.507161 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.507138    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"]
Nov 03 16:17:59.514075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.514047    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-afb990\"/\"premium-w6-afb990-kserve-self-signed-certs\""
Nov 03 16:17:59.609347 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.609301    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:59.609857 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.609839    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-tmp-dir\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.609908 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.609872    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-model-cache\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.609968 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.609952    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-home\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.610006 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.609981    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-dshm\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.610051 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.610025    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw8wg\" (UniqueName: \"kubernetes.io/projected/9d30c691-0b1a-45e5-948b-8826c1173d37-kube-api-access-dw8wg\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.610225 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.610212    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d30c691-0b1a-45e5-948b-8826c1173d37-tls-certs\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.711467 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.711433    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d30c691-0b1a-45e5-948b-8826c1173d37-tls-certs\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.711629 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.711486    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-tmp-dir\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.711629 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.711606    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-model-cache\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.711718 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.711672    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-home\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.711718 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.711710    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-dshm\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.711839 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.711782    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw8wg\" (UniqueName: \"kubernetes.io/projected/9d30c691-0b1a-45e5-948b-8826c1173d37-kube-api-access-dw8wg\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.711915 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.711854    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-tmp-dir\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.711978 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.711942    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-model-cache\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.712031 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.712000    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-home\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.714069 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.714032    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-dshm\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.714256 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.714239    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d30c691-0b1a-45e5-948b-8826c1173d37-tls-certs\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.725598 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.725566    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw8wg\" (UniqueName: \"kubernetes.io/projected/9d30c691-0b1a-45e5-948b-8826c1173d37-kube-api-access-dw8wg\") pod \"premium-w6-afb990-kserve-84c945bdfc-r97wc\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") " pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:17:59.768463 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.768386    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:17:59.807487 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:17:59.807454    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:18:00.090606 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:00.090580    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"]
Nov 03 16:18:00.091854 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:18:00.091821    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d30c691_0b1a_45e5_948b_8826c1173d37.slice/crio-31ce980e922788144ce1a35d8beee93ca6e2c27af9b5889d52739dc880f56f51 WatchSource:0}: Error finding container 31ce980e922788144ce1a35d8beee93ca6e2c27af9b5889d52739dc880f56f51: Status 404 returned error can't find the container with id 31ce980e922788144ce1a35d8beee93ca6e2c27af9b5889d52739dc880f56f51
Nov 03 16:18:00.445261 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:00.445180    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:00.464958 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:00.464915    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" event={"ID":"9d30c691-0b1a-45e5-948b-8826c1173d37","Type":"ContainerStarted","Data":"93c2d3d4511299f8eed5be3b3ac8b0c41595de2d13abadf704e2d5e783d3812d"}
Nov 03 16:18:00.464958 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:00.464959    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" event={"ID":"9d30c691-0b1a-45e5-948b-8826c1173d37","Type":"ContainerStarted","Data":"31ce980e922788144ce1a35d8beee93ca6e2c27af9b5889d52739dc880f56f51"}
Nov 03 16:18:00.489019 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:00.488972    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podStartSLOduration=1.4889583960000001 podStartE2EDuration="1.488958396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:00.486983105 +0000 UTC m=+894.500381252" watchObservedRunningTime="2026-11-03 16:18:00.488958396 +0000 UTC m=+894.502356542"
Nov 03 16:18:00.807853 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:00.807819    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:18:01.144861 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:01.144754    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:01.375879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:01.375851    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:18:01.388753 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:01.388721    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:18:01.482287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:01.482207    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:18:02.023262 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:02.023228    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:03.427174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:03.427135    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:04.287514 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:04.287454    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:04.699554 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:04.699483    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:04.856206 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:04.856171    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:05.072560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:05.072531    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:05.899422 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:05.899385    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:06.721487 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:06.721447    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:07.539134 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:07.539099    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:08.390984 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:08.390954    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:08.998196 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:08.998166    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"]
Nov 03 16:18:09.003645 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.003624    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.007586 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.007558    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d7196a-openshift-default-dockercfg-wxfhk\""
Nov 03 16:18:09.015768 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.015745    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"]
Nov 03 16:18:09.096183 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.096149    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"]
Nov 03 16:18:09.100204 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.100182    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.107923 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.107901    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-69cedd-openshift-default-dockercfg-5bh57\""
Nov 03 16:18:09.118362 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.118334    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"]
Nov 03 16:18:09.122062 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.122038    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.122187 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.122090    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwc7q\" (UniqueName: \"kubernetes.io/projected/05788e58-6105-4a9d-812f-5fcfc45a820c-kube-api-access-nwc7q\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.122252 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.122225    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.122310 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.122278    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.122369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.122347    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.122422 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.122407    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.122488 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.122475    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.122554 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.122542    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05788e58-6105-4a9d-812f-5fcfc45a820c-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.122613 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.122562    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.194908 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.194864    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:09.224042 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224007    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.224249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224049    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.224249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224089    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.224249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224113    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.224249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224136    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.224249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224182    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.224536 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224308    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05788e58-6105-4a9d-812f-5fcfc45a820c-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.224536 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224387    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.224536 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224430    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq68p\" (UniqueName: \"kubernetes.io/projected/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-kube-api-access-mq68p\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.224536 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224470    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.224536 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224498    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.224741 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224561    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.224741 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224569    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwc7q\" (UniqueName: \"kubernetes.io/projected/05788e58-6105-4a9d-812f-5fcfc45a820c-kube-api-access-nwc7q\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.224741 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224642    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.224741 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224682    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.224741 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224718    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.224955 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224748    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.224955 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224776    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.224955 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224836    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.224955 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.224838    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.225245 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.225203    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.225504 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.225478    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.225808 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.225781    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05788e58-6105-4a9d-812f-5fcfc45a820c-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.227360 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.227339    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.227660 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.227643    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.240164 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.240140    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwc7q\" (UniqueName: \"kubernetes.io/projected/05788e58-6105-4a9d-812f-5fcfc45a820c-kube-api-access-nwc7q\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.240490 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.240472    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.318243 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.318214    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:09.325602 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.325583    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.325675 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.325613    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.325713 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.325676    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.325713 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.325692    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.325713 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.325710    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.325869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.325751    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.325869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.325783    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq68p\" (UniqueName: \"kubernetes.io/projected/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-kube-api-access-mq68p\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.325869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.325829    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.326002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.325924    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.326209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.326186    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.326364 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.326337    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.326480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.326366    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.326480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.326408    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.326480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.326453    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.328485 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.328459    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.328563 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.328473    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.352656 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.352626    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.353241 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.353218    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq68p\" (UniqueName: \"kubernetes.io/projected/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-kube-api-access-mq68p\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.379701 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.379650    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:09.414286 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.414253    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:09.607423 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.607392    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"]
Nov 03 16:18:09.608103 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:18:09.608079    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05788e58_6105_4a9d_812f_5fcfc45a820c.slice/crio-6f9e055149a8c0992107faf5adb5e318ebf27b83fd9fe1f5f883ce5eb4bf3f93 WatchSource:0}: Error finding container 6f9e055149a8c0992107faf5adb5e318ebf27b83fd9fe1f5f883ce5eb4bf3f93: Status 404 returned error can't find the container with id 6f9e055149a8c0992107faf5adb5e318ebf27b83fd9fe1f5f883ce5eb4bf3f93
Nov 03 16:18:09.610940 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.610898    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:18:09.611046 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.610981    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:18:09.611046 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.611026    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:18:09.705717 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.705689    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"]
Nov 03 16:18:09.708256 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:18:09.708222    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42754b4_ae9a_41b2_b4e4_0dad7d75a2f3.slice/crio-ebff8b703d8de18fcf96c9e6a1205a6eb71400598ad916bd0af2a41ee4be5226 WatchSource:0}: Error finding container ebff8b703d8de18fcf96c9e6a1205a6eb71400598ad916bd0af2a41ee4be5226: Status 404 returned error can't find the container with id ebff8b703d8de18fcf96c9e6a1205a6eb71400598ad916bd0af2a41ee4be5226
Nov 03 16:18:09.710806 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.710761    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:18:09.710893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.710859    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:18:09.710949 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.710895    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:18:09.807957 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.807921    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:18:09.824879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.824534    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:18:09.931398 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:09.931306    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:10.570085 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:10.570044    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv" event={"ID":"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3","Type":"ContainerStarted","Data":"e636f1aec15879049649c5f555ae75b91d9e3b5727cbf2971acd5fa5eb0ad2ae"}
Nov 03 16:18:10.583935 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:10.583889    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv" event={"ID":"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3","Type":"ContainerStarted","Data":"ebff8b703d8de18fcf96c9e6a1205a6eb71400598ad916bd0af2a41ee4be5226"}
Nov 03 16:18:10.602029 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:10.601876    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t" event={"ID":"05788e58-6105-4a9d-812f-5fcfc45a820c","Type":"ContainerStarted","Data":"7e55e78fdc2a481312d16b2c3ff7739adcda4aa78cdc3be83a9dafa4f4835f53"}
Nov 03 16:18:10.602029 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:10.601976    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t" event={"ID":"05788e58-6105-4a9d-812f-5fcfc45a820c","Type":"ContainerStarted","Data":"6f9e055149a8c0992107faf5adb5e318ebf27b83fd9fe1f5f883ce5eb4bf3f93"}
Nov 03 16:18:10.652819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:10.636039    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv" podStartSLOduration=1.636018846 podStartE2EDuration="1.636018846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:10.631343586 +0000 UTC m=+904.644741732" watchObservedRunningTime="2026-11-03 16:18:10.636018846 +0000 UTC m=+904.649416995"
Nov 03 16:18:10.675505 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:10.675437    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t" podStartSLOduration=2.675397679 podStartE2EDuration="2.675397679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:10.673488505 +0000 UTC m=+904.686886655" watchObservedRunningTime="2026-11-03 16:18:10.675397679 +0000 UTC m=+904.688795825"
Nov 03 16:18:10.709097 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:10.709055    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:10.760467 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:10.760423    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:18:10.997329 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:10.993544    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:11.178930 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:11.173235    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:11.332499 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:11.332468    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:11.354820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:11.337004    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 16:18:11.354820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:11.337060    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t" podUID="05788e58-6105-4a9d-812f-5fcfc45a820c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 16:18:11.419822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:11.418861    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:11.419822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:11.418985    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 16:18:11.419822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:11.419030    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv" podUID="e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 16:18:11.994324 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:11.994253    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:12.319246 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:12.319207    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 16:18:12.319557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:12.319527    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t" podUID="05788e58-6105-4a9d-812f-5fcfc45a820c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 16:18:12.424821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:12.424276    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 16:18:12.424821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:12.424356    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv" podUID="e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 16:18:13.236337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:13.236296    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:13.325822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:13.319156    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 16:18:13.325822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:13.319216    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t" podUID="05788e58-6105-4a9d-812f-5fcfc45a820c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 16:18:13.416224 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:13.416183    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 16:18:13.416501 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:13.416467    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv" podUID="e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 16:18:14.247828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:14.245296    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:14.320819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:14.319782    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 16:18:14.320819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:14.319857    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t" podUID="05788e58-6105-4a9d-812f-5fcfc45a820c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 16:18:14.419821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:14.415227    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 16:18:14.419821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:14.415286    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv" podUID="e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 16:18:15.046821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:15.046564    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:15.324181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:15.324092    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 16:18:15.324744 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:15.324711    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t" podUID="05788e58-6105-4a9d-812f-5fcfc45a820c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 16:18:15.422823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:15.416273    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 16:18:15.422823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:15.416338    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv" podUID="e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 16:18:15.879852 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:15.879771    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:16.319217 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:16.319176    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 16:18:16.319512 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:16.319480    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t" podUID="05788e58-6105-4a9d-812f-5fcfc45a820c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 16:18:16.415332 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:16.415287    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 16:18:16.415926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:16.415876    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv" podUID="e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 16:18:16.609323 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:16.609226    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:17.329818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:17.329439    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:17.627874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:17.627234    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:17.627874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:17.627765    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:17.634819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:17.632064    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:18:17.895209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:17.895133    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:17.895486 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:17.895459    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:17.896179 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:17.896164    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:18:18.308067 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:18.308036    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:19.235263 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:19.235224    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:20.014509 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:20.014474    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:20.813018 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:20.812978    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:21.621041 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:21.621010    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:22.312546 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:22.312513    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:22.993277 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:22.993246    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:23.931694 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:23.931661    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:24.750424 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:24.750388    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:25.745379 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:25.745347    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:27.040683 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:27.040646    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:28.202889 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:28.202837    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:28.796482 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:18:28.796447    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:18:29.308562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:29.308305    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:30.381291 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:30.381250    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:31.218119 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:31.218066    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:32.031570 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:32.031532    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:33.009231 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:33.009185    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:33.826842 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:33.826790    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:34.718217 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:34.718166    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:35.417931 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:35.417880    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:36.406364 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:36.406327    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:37.420236 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:37.420192    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:38.419337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:38.419307    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:39.598276 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:39.598241    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:40.105505 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:40.105469    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:40.792808 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:40.792763    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"]
Nov 03 16:18:40.798643 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:40.798626    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"
Nov 03 16:18:40.808073 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:40.808042    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:18:40.808073 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:40.808060    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:18:40.810869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:40.810844    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"]
Nov 03 16:18:40.866318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:40.866290    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh2sn\" (UniqueName: \"kubernetes.io/projected/6df4e02c-e4a4-417b-aec0-f08f98c10860-kube-api-access-rh2sn\") pod \"maas-api-revoke-sub-ed386566-pr6jk\" (UID: \"6df4e02c-e4a4-417b-aec0-f08f98c10860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"
Nov 03 16:18:40.866470 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:40.866338    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df4e02c-e4a4-417b-aec0-f08f98c10860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed386566-pr6jk\" (UID: \"6df4e02c-e4a4-417b-aec0-f08f98c10860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"
Nov 03 16:18:40.866470 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:40.866455    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df4e02c-e4a4-417b-aec0-f08f98c10860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed386566-pr6jk\" (UID: \"6df4e02c-e4a4-417b-aec0-f08f98c10860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"
Nov 03 16:18:40.967412 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:40.967373    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh2sn\" (UniqueName: \"kubernetes.io/projected/6df4e02c-e4a4-417b-aec0-f08f98c10860-kube-api-access-rh2sn\") pod \"maas-api-revoke-sub-ed386566-pr6jk\" (UID: \"6df4e02c-e4a4-417b-aec0-f08f98c10860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"
Nov 03 16:18:40.967606 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:40.967440    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df4e02c-e4a4-417b-aec0-f08f98c10860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed386566-pr6jk\" (UID: \"6df4e02c-e4a4-417b-aec0-f08f98c10860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"
Nov 03 16:18:40.967606 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:40.967536    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df4e02c-e4a4-417b-aec0-f08f98c10860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed386566-pr6jk\" (UID: \"6df4e02c-e4a4-417b-aec0-f08f98c10860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"
Nov 03 16:18:40.968304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:40.968272    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df4e02c-e4a4-417b-aec0-f08f98c10860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed386566-pr6jk\" (UID: \"6df4e02c-e4a4-417b-aec0-f08f98c10860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"
Nov 03 16:18:40.970349 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:40.970321    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df4e02c-e4a4-417b-aec0-f08f98c10860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed386566-pr6jk\" (UID: \"6df4e02c-e4a4-417b-aec0-f08f98c10860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"
Nov 03 16:18:40.983187 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:40.983162    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh2sn\" (UniqueName: \"kubernetes.io/projected/6df4e02c-e4a4-417b-aec0-f08f98c10860-kube-api-access-rh2sn\") pod \"maas-api-revoke-sub-ed386566-pr6jk\" (UID: \"6df4e02c-e4a4-417b-aec0-f08f98c10860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"
Nov 03 16:18:41.003229 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:41.003194    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:41.115440 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:41.115370    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"
Nov 03 16:18:41.436512 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:41.436478    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"]
Nov 03 16:18:41.436948 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:18:41.436911    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df4e02c_e4a4_417b_aec0_f08f98c10860.slice/crio-5ffb59381576f0d0c1204fc32623ca15349337336efc5ceac7ba9df26707b634 WatchSource:0}: Error finding container 5ffb59381576f0d0c1204fc32623ca15349337336efc5ceac7ba9df26707b634: Status 404 returned error can't find the container with id 5ffb59381576f0d0c1204fc32623ca15349337336efc5ceac7ba9df26707b634
Nov 03 16:18:41.851518 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:41.851154    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:41.895959 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:41.895921    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk" event={"ID":"6df4e02c-e4a4-417b-aec0-f08f98c10860","Type":"ContainerStarted","Data":"be04111f0641df101b59b73a2add9c4e5c6d7d00f03a11de8c3e8cd4e1f2a247"}
Nov 03 16:18:41.896159 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:41.895983    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk" event={"ID":"6df4e02c-e4a4-417b-aec0-f08f98c10860","Type":"ContainerStarted","Data":"5ffb59381576f0d0c1204fc32623ca15349337336efc5ceac7ba9df26707b634"}
Nov 03 16:18:41.917309 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:41.917250    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk" podStartSLOduration=1.917230327 podStartE2EDuration="1.917230327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:41.914534291 +0000 UTC m=+935.927932437" watchObservedRunningTime="2026-11-03 16:18:41.917230327 +0000 UTC m=+935.930628474"
Nov 03 16:18:42.834837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:42.834764    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:42.903462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:42.903429    6506 generic.go:356] "Generic (PLEG): container finished" podID="6df4e02c-e4a4-417b-aec0-f08f98c10860" containerID="be04111f0641df101b59b73a2add9c4e5c6d7d00f03a11de8c3e8cd4e1f2a247" exitCode=0
Nov 03 16:18:42.903932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:42.903515    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk" event={"ID":"6df4e02c-e4a4-417b-aec0-f08f98c10860","Type":"ContainerDied","Data":"be04111f0641df101b59b73a2add9c4e5c6d7d00f03a11de8c3e8cd4e1f2a247"}
Nov 03 16:18:43.738506 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:43.738472    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:44.043909 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:44.043884    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"
Nov 03 16:18:44.108403 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:44.108368    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df4e02c-e4a4-417b-aec0-f08f98c10860-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df4e02c-e4a4-417b-aec0-f08f98c10860-maas-api-openshift-ca\") pod \"6df4e02c-e4a4-417b-aec0-f08f98c10860\" (UID: \"6df4e02c-e4a4-417b-aec0-f08f98c10860\") "
Nov 03 16:18:44.108608 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:44.108447    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df4e02c-e4a4-417b-aec0-f08f98c10860-kube-api-access-rh2sn\" (UniqueName: \"kubernetes.io/projected/6df4e02c-e4a4-417b-aec0-f08f98c10860-kube-api-access-rh2sn\") pod \"6df4e02c-e4a4-417b-aec0-f08f98c10860\" (UID: \"6df4e02c-e4a4-417b-aec0-f08f98c10860\") "
Nov 03 16:18:44.108726 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:44.108671    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6df4e02c-e4a4-417b-aec0-f08f98c10860-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df4e02c-e4a4-417b-aec0-f08f98c10860-maas-api-xks-ca\") pod \"6df4e02c-e4a4-417b-aec0-f08f98c10860\" (UID: \"6df4e02c-e4a4-417b-aec0-f08f98c10860\") "
Nov 03 16:18:44.108833 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:44.108761    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df4e02c-e4a4-417b-aec0-f08f98c10860-maas-api-openshift-ca" pod "6df4e02c-e4a4-417b-aec0-f08f98c10860" (UID: "6df4e02c-e4a4-417b-aec0-f08f98c10860"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:18:44.109104 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:44.109075    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df4e02c-e4a4-417b-aec0-f08f98c10860-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:44.111337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:44.111309    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df4e02c-e4a4-417b-aec0-f08f98c10860-maas-api-xks-ca" pod "6df4e02c-e4a4-417b-aec0-f08f98c10860" (UID: "6df4e02c-e4a4-417b-aec0-f08f98c10860"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:18:44.112258 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:44.112226    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df4e02c-e4a4-417b-aec0-f08f98c10860-kube-api-access-rh2sn" pod "6df4e02c-e4a4-417b-aec0-f08f98c10860" (UID: "6df4e02c-e4a4-417b-aec0-f08f98c10860"). InnerVolumeSpecName "kube-api-access-rh2sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:18:44.210412 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:44.210376    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df4e02c-e4a4-417b-aec0-f08f98c10860-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:44.210412 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:44.210407    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh2sn\" (UniqueName: \"kubernetes.io/projected/6df4e02c-e4a4-417b-aec0-f08f98c10860-kube-api-access-rh2sn\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:44.643612 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:44.643571    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:44.917901 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:44.917785    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk" event={"ID":"6df4e02c-e4a4-417b-aec0-f08f98c10860","Type":"ContainerDied","Data":"5ffb59381576f0d0c1204fc32623ca15349337336efc5ceac7ba9df26707b634"}
Nov 03 16:18:44.917901 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:44.917838    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"
Nov 03 16:18:44.917901 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:44.917868    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ffb59381576f0d0c1204fc32623ca15349337336efc5ceac7ba9df26707b634"
Nov 03 16:18:45.483804 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.483755    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"]
Nov 03 16:18:45.484390 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.484372    6506 cpu_manager.go:414] "containerMap: removing container" podUID="6df4e02c-e4a4-417b-aec0-f08f98c10860" containerName="revoke-keys"
Nov 03 16:18:45.484451 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.484393    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df4e02c-e4a4-417b-aec0-f08f98c10860" containerName="revoke-keys"
Nov 03 16:18:45.484511 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.484502    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df4e02c-e4a4-417b-aec0-f08f98c10860" containerName="revoke-keys"
Nov 03 16:18:45.487737 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.487721    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"
Nov 03 16:18:45.497156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.497131    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:18:45.497336 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.497313    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:18:45.497919 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.497897    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"]
Nov 03 16:18:45.531592 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.531554    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d0cdbc6-6szhc\" (UID: \"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"
Nov 03 16:18:45.531874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.531648    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d0cdbc6-6szhc\" (UID: \"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"
Nov 03 16:18:45.531874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.531747    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pfdl\" (UniqueName: \"kubernetes.io/projected/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-kube-api-access-6pfdl\") pod \"maas-api-revoke-sub-4d0cdbc6-6szhc\" (UID: \"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"
Nov 03 16:18:45.633181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.633150    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d0cdbc6-6szhc\" (UID: \"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"
Nov 03 16:18:45.633381 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.633212    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d0cdbc6-6szhc\" (UID: \"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"
Nov 03 16:18:45.633381 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.633274    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pfdl\" (UniqueName: \"kubernetes.io/projected/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-kube-api-access-6pfdl\") pod \"maas-api-revoke-sub-4d0cdbc6-6szhc\" (UID: \"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"
Nov 03 16:18:45.633937 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.633913    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d0cdbc6-6szhc\" (UID: \"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"
Nov 03 16:18:45.635714 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.635696    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d0cdbc6-6szhc\" (UID: \"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"
Nov 03 16:18:45.646104 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.646079    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pfdl\" (UniqueName: \"kubernetes.io/projected/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-kube-api-access-6pfdl\") pod \"maas-api-revoke-sub-4d0cdbc6-6szhc\" (UID: \"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"
Nov 03 16:18:45.802389 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.802364    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"
Nov 03 16:18:45.807583 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:45.807558    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:46.080312 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:46.080282    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"]
Nov 03 16:18:46.081544 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:18:46.081517    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c2e0c1_fbbb_4adf_bcae_07fc9ecb4746.slice/crio-fa7d23baf1c7f1f5c7ad21898cc2251dff24508d5eeeae4760d79b7704f5a6f8 WatchSource:0}: Error finding container fa7d23baf1c7f1f5c7ad21898cc2251dff24508d5eeeae4760d79b7704f5a6f8: Status 404 returned error can't find the container with id fa7d23baf1c7f1f5c7ad21898cc2251dff24508d5eeeae4760d79b7704f5a6f8
Nov 03 16:18:46.409611 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:46.409542    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:46.931714 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:46.931679    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc" event={"ID":"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746","Type":"ContainerStarted","Data":"88c47481e33da655eebc0d4edc6263cb29ee03de259739dc94e7e9639e8e3035"}
Nov 03 16:18:46.932123 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:46.931730    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc" event={"ID":"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746","Type":"ContainerStarted","Data":"fa7d23baf1c7f1f5c7ad21898cc2251dff24508d5eeeae4760d79b7704f5a6f8"}
Nov 03 16:18:46.954425 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:46.954379    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc" podStartSLOduration=1.954366663 podStartE2EDuration="1.954366663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:46.9525304 +0000 UTC m=+940.965928548" watchObservedRunningTime="2026-11-03 16:18:46.954366663 +0000 UTC m=+940.967764837"
Nov 03 16:18:47.131841 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:47.131770    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:47.814566 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:47.814521    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:47.939074 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:47.939038    6506 generic.go:356] "Generic (PLEG): container finished" podID="a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746" containerID="88c47481e33da655eebc0d4edc6263cb29ee03de259739dc94e7e9639e8e3035" exitCode=0
Nov 03 16:18:47.939487 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:47.939119    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc" event={"ID":"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746","Type":"ContainerDied","Data":"88c47481e33da655eebc0d4edc6263cb29ee03de259739dc94e7e9639e8e3035"}
Nov 03 16:18:48.813604 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:48.813561    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:49.132999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:49.132970    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"
Nov 03 16:18:49.178981 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:49.178946    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-maas-api-openshift-ca\") pod \"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746\" (UID: \"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746\") "
Nov 03 16:18:49.179136 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:49.179077    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-maas-api-xks-ca\") pod \"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746\" (UID: \"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746\") "
Nov 03 16:18:49.179245 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:49.179228    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-kube-api-access-6pfdl\" (UniqueName: \"kubernetes.io/projected/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-kube-api-access-6pfdl\") pod \"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746\" (UID: \"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746\") "
Nov 03 16:18:49.179409 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:49.179382    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-maas-api-openshift-ca" pod "a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746" (UID: "a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:18:49.179753 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:49.179727    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:49.182066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:49.182026    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-maas-api-xks-ca" pod "a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746" (UID: "a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:18:49.182066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:49.182052    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-kube-api-access-6pfdl" pod "a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746" (UID: "a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746"). InnerVolumeSpecName "kube-api-access-6pfdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:18:49.280755 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:49.280727    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:49.280755 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:49.280753    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pfdl\" (UniqueName: \"kubernetes.io/projected/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746-kube-api-access-6pfdl\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:49.911882 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:49.911841    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:49.954128 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:49.954095    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"
Nov 03 16:18:49.954318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:49.954136    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc" event={"ID":"a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746","Type":"ContainerDied","Data":"fa7d23baf1c7f1f5c7ad21898cc2251dff24508d5eeeae4760d79b7704f5a6f8"}
Nov 03 16:18:49.954318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:49.954215    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa7d23baf1c7f1f5c7ad21898cc2251dff24508d5eeeae4760d79b7704f5a6f8"
Nov 03 16:18:50.733905 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:50.733869    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:51.521966 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:51.521931    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:52.305414 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.305380    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"]
Nov 03 16:18:52.305986 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.305970    6506 cpu_manager.go:414] "containerMap: removing container" podUID="a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746" containerName="revoke-keys"
Nov 03 16:18:52.306040 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.305987    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746" containerName="revoke-keys"
Nov 03 16:18:52.306073 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.306064    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746" containerName="revoke-keys"
Nov 03 16:18:52.309262 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.309248    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"
Nov 03 16:18:52.320320 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.320293    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"]
Nov 03 16:18:52.320781 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.320760    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:18:52.320919 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.320772    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:18:52.416329 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.416294    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070f7662-d077-4fa3-bcf7-59ceb9eecb04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41438547-lmlcx\" (UID: \"070f7662-d077-4fa3-bcf7-59ceb9eecb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"
Nov 03 16:18:52.416591 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.416355    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltbxw\" (UniqueName: \"kubernetes.io/projected/070f7662-d077-4fa3-bcf7-59ceb9eecb04-kube-api-access-ltbxw\") pod \"maas-api-revoke-sub-41438547-lmlcx\" (UID: \"070f7662-d077-4fa3-bcf7-59ceb9eecb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"
Nov 03 16:18:52.416591 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.416471    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070f7662-d077-4fa3-bcf7-59ceb9eecb04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41438547-lmlcx\" (UID: \"070f7662-d077-4fa3-bcf7-59ceb9eecb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"
Nov 03 16:18:52.441490 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.441439    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:52.517135 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.517100    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070f7662-d077-4fa3-bcf7-59ceb9eecb04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41438547-lmlcx\" (UID: \"070f7662-d077-4fa3-bcf7-59ceb9eecb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"
Nov 03 16:18:52.517292 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.517269    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070f7662-d077-4fa3-bcf7-59ceb9eecb04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41438547-lmlcx\" (UID: \"070f7662-d077-4fa3-bcf7-59ceb9eecb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"
Nov 03 16:18:52.517341 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.517320    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltbxw\" (UniqueName: \"kubernetes.io/projected/070f7662-d077-4fa3-bcf7-59ceb9eecb04-kube-api-access-ltbxw\") pod \"maas-api-revoke-sub-41438547-lmlcx\" (UID: \"070f7662-d077-4fa3-bcf7-59ceb9eecb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"
Nov 03 16:18:52.517921 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.517898    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070f7662-d077-4fa3-bcf7-59ceb9eecb04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41438547-lmlcx\" (UID: \"070f7662-d077-4fa3-bcf7-59ceb9eecb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"
Nov 03 16:18:52.519648 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.519628    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070f7662-d077-4fa3-bcf7-59ceb9eecb04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41438547-lmlcx\" (UID: \"070f7662-d077-4fa3-bcf7-59ceb9eecb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"
Nov 03 16:18:52.530193 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.530168    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltbxw\" (UniqueName: \"kubernetes.io/projected/070f7662-d077-4fa3-bcf7-59ceb9eecb04-kube-api-access-ltbxw\") pod \"maas-api-revoke-sub-41438547-lmlcx\" (UID: \"070f7662-d077-4fa3-bcf7-59ceb9eecb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"
Nov 03 16:18:52.624417 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.624351    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"
Nov 03 16:18:52.912850 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.912450    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"]
Nov 03 16:18:52.917869 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:18:52.914890    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070f7662_d077_4fa3_bcf7_59ceb9eecb04.slice/crio-3b11d624ae8f428c33625b318f7d7cbeccbe1f3fb5b5f9f18e2ae05b7fd0f95f WatchSource:0}: Error finding container 3b11d624ae8f428c33625b318f7d7cbeccbe1f3fb5b5f9f18e2ae05b7fd0f95f: Status 404 returned error can't find the container with id 3b11d624ae8f428c33625b318f7d7cbeccbe1f3fb5b5f9f18e2ae05b7fd0f95f
Nov 03 16:18:52.975203 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:52.975171    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx" event={"ID":"070f7662-d077-4fa3-bcf7-59ceb9eecb04","Type":"ContainerStarted","Data":"3b11d624ae8f428c33625b318f7d7cbeccbe1f3fb5b5f9f18e2ae05b7fd0f95f"}
Nov 03 16:18:53.437478 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:53.437441    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:53.912749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:53.912719    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"]
Nov 03 16:18:53.916594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:53.916571    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:53.920756 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:53.920735    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4a19cd-openshift-default-dockercfg-h4cvm\""
Nov 03 16:18:53.937105 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:53.937073    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"]
Nov 03 16:18:53.983282 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:53.983197    6506 generic.go:356] "Generic (PLEG): container finished" podID="070f7662-d077-4fa3-bcf7-59ceb9eecb04" containerID="19e4a6e549cc3576e37e369388b3d7542e2b46a4438b3bea0a6a7d3f5abffd1f" exitCode=0
Nov 03 16:18:53.983435 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:53.983307    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx" event={"ID":"070f7662-d077-4fa3-bcf7-59ceb9eecb04","Type":"ContainerDied","Data":"19e4a6e549cc3576e37e369388b3d7542e2b46a4438b3bea0a6a7d3f5abffd1f"}
Nov 03 16:18:54.042916 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.042882    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.043043 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.042953    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.043043 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.043011    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.043138 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.043051    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27399426-17f7-4ea7-aba6-12338b82c6f3-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.043138 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.043088    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.043215 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.043134    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.043215 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.043174    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.043335 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.043315    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf9jp\" (UniqueName: \"kubernetes.io/projected/27399426-17f7-4ea7-aba6-12338b82c6f3-kube-api-access-wf9jp\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.043399 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.043351    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.144819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.144468    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.144819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.144620    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf9jp\" (UniqueName: \"kubernetes.io/projected/27399426-17f7-4ea7-aba6-12338b82c6f3-kube-api-access-wf9jp\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.144819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.144652    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.144819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.144718    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.144819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.144782    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.145492 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.144831    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.145492 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.144902    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27399426-17f7-4ea7-aba6-12338b82c6f3-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.145492 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.144953    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.145492 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.144977    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.145492 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.145475    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.146381 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.146356    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.147038 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.146959    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.147725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.147703    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.148531 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.147700    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27399426-17f7-4ea7-aba6-12338b82c6f3-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.155978 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.155954    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.157036 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.157016    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.164576 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.164548    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf9jp\" (UniqueName: \"kubernetes.io/projected/27399426-17f7-4ea7-aba6-12338b82c6f3-kube-api-access-wf9jp\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.165531 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.165507    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.231321 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.231294    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:54.322855 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.314173    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:54.547930 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:18:54.547882    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27399426_17f7_4ea7_aba6_12338b82c6f3.slice/crio-fe7ea970654d1da6ea4029883c28ec6f52dcabc9aaa935be9906bd4cc9cf78d8 WatchSource:0}: Error finding container fe7ea970654d1da6ea4029883c28ec6f52dcabc9aaa935be9906bd4cc9cf78d8: Status 404 returned error can't find the container with id fe7ea970654d1da6ea4029883c28ec6f52dcabc9aaa935be9906bd4cc9cf78d8
Nov 03 16:18:54.550554 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.550516    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:18:54.550647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.550590    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:18:54.550647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.550623    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:18:54.552089 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.552066    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"]
Nov 03 16:18:54.999882 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.998532    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc" event={"ID":"27399426-17f7-4ea7-aba6-12338b82c6f3","Type":"ContainerStarted","Data":"6af2f3ffed58b75a271a821521541f45e11b196e361df4762fd1995408c88343"}
Nov 03 16:18:54.999882 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:54.998594    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc" event={"ID":"27399426-17f7-4ea7-aba6-12338b82c6f3","Type":"ContainerStarted","Data":"fe7ea970654d1da6ea4029883c28ec6f52dcabc9aaa935be9906bd4cc9cf78d8"}
Nov 03 16:18:55.035170 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:55.035110    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc" podStartSLOduration=2.035094952 podStartE2EDuration="2.035094952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:55.032254536 +0000 UTC m=+949.045652681" watchObservedRunningTime="2026-11-03 16:18:55.035094952 +0000 UTC m=+949.048493097"
Nov 03 16:18:55.179115 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:55.179091    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"
Nov 03 16:18:55.237680 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:55.237362    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 16:18:55.237680 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:55.237408    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:18:55.237680 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:55.237421    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc" podUID="27399426-17f7-4ea7-aba6-12338b82c6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 16:18:55.260375 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:55.260293    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070f7662-d077-4fa3-bcf7-59ceb9eecb04-kube-api-access-ltbxw\" (UniqueName: \"kubernetes.io/projected/070f7662-d077-4fa3-bcf7-59ceb9eecb04-kube-api-access-ltbxw\") pod \"070f7662-d077-4fa3-bcf7-59ceb9eecb04\" (UID: \"070f7662-d077-4fa3-bcf7-59ceb9eecb04\") "
Nov 03 16:18:55.260519 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:55.260484    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/070f7662-d077-4fa3-bcf7-59ceb9eecb04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070f7662-d077-4fa3-bcf7-59ceb9eecb04-maas-api-openshift-ca\") pod \"070f7662-d077-4fa3-bcf7-59ceb9eecb04\" (UID: \"070f7662-d077-4fa3-bcf7-59ceb9eecb04\") "
Nov 03 16:18:55.260586 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:55.260517    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/070f7662-d077-4fa3-bcf7-59ceb9eecb04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070f7662-d077-4fa3-bcf7-59ceb9eecb04-maas-api-xks-ca\") pod \"070f7662-d077-4fa3-bcf7-59ceb9eecb04\" (UID: \"070f7662-d077-4fa3-bcf7-59ceb9eecb04\") "
Nov 03 16:18:55.261357 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:55.261284    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070f7662-d077-4fa3-bcf7-59ceb9eecb04-maas-api-openshift-ca" pod "070f7662-d077-4fa3-bcf7-59ceb9eecb04" (UID: "070f7662-d077-4fa3-bcf7-59ceb9eecb04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:18:55.262726 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:55.262698    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/070f7662-d077-4fa3-bcf7-59ceb9eecb04-maas-api-xks-ca" pod "070f7662-d077-4fa3-bcf7-59ceb9eecb04" (UID: "070f7662-d077-4fa3-bcf7-59ceb9eecb04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:18:55.262976 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:55.262956    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070f7662-d077-4fa3-bcf7-59ceb9eecb04-kube-api-access-ltbxw" pod "070f7662-d077-4fa3-bcf7-59ceb9eecb04" (UID: "070f7662-d077-4fa3-bcf7-59ceb9eecb04"). InnerVolumeSpecName "kube-api-access-ltbxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:18:55.345023 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:55.344985    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:55.361756 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:55.361730    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070f7662-d077-4fa3-bcf7-59ceb9eecb04-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:55.361756 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:55.361755    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltbxw\" (UniqueName: \"kubernetes.io/projected/070f7662-d077-4fa3-bcf7-59ceb9eecb04-kube-api-access-ltbxw\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:55.361932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:55.361768    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070f7662-d077-4fa3-bcf7-59ceb9eecb04-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:56.004597 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:56.004512    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"
Nov 03 16:18:56.004597 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:56.004551    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx" event={"ID":"070f7662-d077-4fa3-bcf7-59ceb9eecb04","Type":"ContainerDied","Data":"3b11d624ae8f428c33625b318f7d7cbeccbe1f3fb5b5f9f18e2ae05b7fd0f95f"}
Nov 03 16:18:56.004597 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:56.004595    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b11d624ae8f428c33625b318f7d7cbeccbe1f3fb5b5f9f18e2ae05b7fd0f95f"
Nov 03 16:18:56.233042 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:56.233001    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 16:18:56.233251 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:56.233067    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc" podUID="27399426-17f7-4ea7-aba6-12338b82c6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 16:18:56.289731 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:56.289700    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:57.035864 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.034866    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:57.232818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.232270    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 16:18:57.232818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.232352    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc" podUID="27399426-17f7-4ea7-aba6-12338b82c6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 16:18:57.635073 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.635033    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"]
Nov 03 16:18:57.635610 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.635595    6506 cpu_manager.go:414] "containerMap: removing container" podUID="070f7662-d077-4fa3-bcf7-59ceb9eecb04" containerName="revoke-keys"
Nov 03 16:18:57.635650 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.635612    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070f7662-d077-4fa3-bcf7-59ceb9eecb04" containerName="revoke-keys"
Nov 03 16:18:57.635739 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.635729    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="070f7662-d077-4fa3-bcf7-59ceb9eecb04" containerName="revoke-keys"
Nov 03 16:18:57.639315 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.639296    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"
Nov 03 16:18:57.649828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.649807    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:18:57.650012 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.649994    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:18:57.650374 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.650357    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"]
Nov 03 16:18:57.678484 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.678247    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"]
Nov 03 16:18:57.682728 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.682695    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"]
Nov 03 16:18:57.682901 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.682851    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"
Nov 03 16:18:57.690511 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.690485    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r2m5\" (UniqueName: \"kubernetes.io/projected/df53eb88-f023-4084-9e6a-eecbada27f7a-kube-api-access-8r2m5\") pod \"maas-api-revoke-sub-6e857453-x2c4b\" (UID: \"df53eb88-f023-4084-9e6a-eecbada27f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"
Nov 03 16:18:57.690622 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.690537    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df53eb88-f023-4084-9e6a-eecbada27f7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e857453-x2c4b\" (UID: \"df53eb88-f023-4084-9e6a-eecbada27f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"
Nov 03 16:18:57.690622 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.690591    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df53eb88-f023-4084-9e6a-eecbada27f7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e857453-x2c4b\" (UID: \"df53eb88-f023-4084-9e6a-eecbada27f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"
Nov 03 16:18:57.791224 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.791194    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt2fl\" (UniqueName: \"kubernetes.io/projected/e94174aa-83ef-4037-861a-81dfbb6894d1-kube-api-access-nt2fl\") pod \"maas-api-revoke-sub-b2956e28-8ccs9\" (UID: \"e94174aa-83ef-4037-861a-81dfbb6894d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"
Nov 03 16:18:57.791400 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.791253    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94174aa-83ef-4037-861a-81dfbb6894d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2956e28-8ccs9\" (UID: \"e94174aa-83ef-4037-861a-81dfbb6894d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"
Nov 03 16:18:57.791400 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.791341    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r2m5\" (UniqueName: \"kubernetes.io/projected/df53eb88-f023-4084-9e6a-eecbada27f7a-kube-api-access-8r2m5\") pod \"maas-api-revoke-sub-6e857453-x2c4b\" (UID: \"df53eb88-f023-4084-9e6a-eecbada27f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"
Nov 03 16:18:57.791400 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.791388    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df53eb88-f023-4084-9e6a-eecbada27f7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e857453-x2c4b\" (UID: \"df53eb88-f023-4084-9e6a-eecbada27f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"
Nov 03 16:18:57.791531 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.791444    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94174aa-83ef-4037-861a-81dfbb6894d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2956e28-8ccs9\" (UID: \"e94174aa-83ef-4037-861a-81dfbb6894d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"
Nov 03 16:18:57.791531 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.791476    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df53eb88-f023-4084-9e6a-eecbada27f7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e857453-x2c4b\" (UID: \"df53eb88-f023-4084-9e6a-eecbada27f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"
Nov 03 16:18:57.792093 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.792074    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df53eb88-f023-4084-9e6a-eecbada27f7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e857453-x2c4b\" (UID: \"df53eb88-f023-4084-9e6a-eecbada27f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"
Nov 03 16:18:57.793930 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.793911    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df53eb88-f023-4084-9e6a-eecbada27f7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e857453-x2c4b\" (UID: \"df53eb88-f023-4084-9e6a-eecbada27f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"
Nov 03 16:18:57.804023 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.804005    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r2m5\" (UniqueName: \"kubernetes.io/projected/df53eb88-f023-4084-9e6a-eecbada27f7a-kube-api-access-8r2m5\") pod \"maas-api-revoke-sub-6e857453-x2c4b\" (UID: \"df53eb88-f023-4084-9e6a-eecbada27f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"
Nov 03 16:18:57.893148 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.893069    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt2fl\" (UniqueName: \"kubernetes.io/projected/e94174aa-83ef-4037-861a-81dfbb6894d1-kube-api-access-nt2fl\") pod \"maas-api-revoke-sub-b2956e28-8ccs9\" (UID: \"e94174aa-83ef-4037-861a-81dfbb6894d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"
Nov 03 16:18:57.893290 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.893171    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94174aa-83ef-4037-861a-81dfbb6894d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2956e28-8ccs9\" (UID: \"e94174aa-83ef-4037-861a-81dfbb6894d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"
Nov 03 16:18:57.893330 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.893283    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94174aa-83ef-4037-861a-81dfbb6894d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2956e28-8ccs9\" (UID: \"e94174aa-83ef-4037-861a-81dfbb6894d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"
Nov 03 16:18:57.894004 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.893981    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94174aa-83ef-4037-861a-81dfbb6894d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2956e28-8ccs9\" (UID: \"e94174aa-83ef-4037-861a-81dfbb6894d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"
Nov 03 16:18:57.895714 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.895693    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94174aa-83ef-4037-861a-81dfbb6894d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2956e28-8ccs9\" (UID: \"e94174aa-83ef-4037-861a-81dfbb6894d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"
Nov 03 16:18:57.905861 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.905838    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt2fl\" (UniqueName: \"kubernetes.io/projected/e94174aa-83ef-4037-861a-81dfbb6894d1-kube-api-access-nt2fl\") pod \"maas-api-revoke-sub-b2956e28-8ccs9\" (UID: \"e94174aa-83ef-4037-861a-81dfbb6894d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"
Nov 03 16:18:57.954420 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.954393    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"
Nov 03 16:18:57.996359 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.996288    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:57.998023 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:57.998001    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"
Nov 03 16:18:58.232819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:58.232474    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 16:18:58.232819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:58.232529    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc" podUID="27399426-17f7-4ea7-aba6-12338b82c6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 16:18:58.303314 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:58.303286    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"]
Nov 03 16:18:58.304048 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:18:58.304026    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf53eb88_f023_4084_9e6a_eecbada27f7a.slice/crio-36aeb831dfbd0463755dad9bbc822d6e5ae4cd304fcbd5eb6eec644ea094af2b WatchSource:0}: Error finding container 36aeb831dfbd0463755dad9bbc822d6e5ae4cd304fcbd5eb6eec644ea094af2b: Status 404 returned error can't find the container with id 36aeb831dfbd0463755dad9bbc822d6e5ae4cd304fcbd5eb6eec644ea094af2b
Nov 03 16:18:58.371591 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:58.371567    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"]
Nov 03 16:18:58.373087 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:18:58.373058    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94174aa_83ef_4037_861a_81dfbb6894d1.slice/crio-1dec1a710b017b1f25c25e319c7c136942ec357379bc8b3e5d27c9d6b5334ba6 WatchSource:0}: Error finding container 1dec1a710b017b1f25c25e319c7c136942ec357379bc8b3e5d27c9d6b5334ba6: Status 404 returned error can't find the container with id 1dec1a710b017b1f25c25e319c7c136942ec357379bc8b3e5d27c9d6b5334ba6
Nov 03 16:18:58.830275 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:58.830179    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:18:58.866392 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:18:58.866322    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:18:59.063178 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:59.063139    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b" event={"ID":"df53eb88-f023-4084-9e6a-eecbada27f7a","Type":"ContainerStarted","Data":"6c6a7dc8c0221371cf7d33a6ec16b8640d739728d23df5185f135b72f1e4b68e"}
Nov 03 16:18:59.063373 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:59.063200    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b" event={"ID":"df53eb88-f023-4084-9e6a-eecbada27f7a","Type":"ContainerStarted","Data":"36aeb831dfbd0463755dad9bbc822d6e5ae4cd304fcbd5eb6eec644ea094af2b"}
Nov 03 16:18:59.065113 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:59.065076    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9" event={"ID":"e94174aa-83ef-4037-861a-81dfbb6894d1","Type":"ContainerStarted","Data":"48e0b5c997d48f979e428bb94d9c148140c082fee844f8727e1533b026cb3ffb"}
Nov 03 16:18:59.065240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:59.065135    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9" event={"ID":"e94174aa-83ef-4037-861a-81dfbb6894d1","Type":"ContainerStarted","Data":"1dec1a710b017b1f25c25e319c7c136942ec357379bc8b3e5d27c9d6b5334ba6"}
Nov 03 16:18:59.091365 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:59.091319    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b" podStartSLOduration=2.091307024 podStartE2EDuration="2.091307024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:59.089736231 +0000 UTC m=+953.103134378" watchObservedRunningTime="2026-11-03 16:18:59.091307024 +0000 UTC m=+953.104705170"
Nov 03 16:18:59.237177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:59.237038    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 16:18:59.237177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:59.237101    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc" podUID="27399426-17f7-4ea7-aba6-12338b82c6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 16:18:59.635874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:18:59.626783    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:00.073240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:00.073203    6506 generic.go:356] "Generic (PLEG): container finished" podID="e94174aa-83ef-4037-861a-81dfbb6894d1" containerID="48e0b5c997d48f979e428bb94d9c148140c082fee844f8727e1533b026cb3ffb" exitCode=0
Nov 03 16:19:00.073420 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:00.073300    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9" event={"ID":"e94174aa-83ef-4037-861a-81dfbb6894d1","Type":"ContainerDied","Data":"48e0b5c997d48f979e428bb94d9c148140c082fee844f8727e1533b026cb3ffb"}
Nov 03 16:19:00.075035 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:00.075005    6506 generic.go:356] "Generic (PLEG): container finished" podID="df53eb88-f023-4084-9e6a-eecbada27f7a" containerID="6c6a7dc8c0221371cf7d33a6ec16b8640d739728d23df5185f135b72f1e4b68e" exitCode=0
Nov 03 16:19:00.075165 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:00.075072    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b" event={"ID":"df53eb88-f023-4084-9e6a-eecbada27f7a","Type":"ContainerDied","Data":"6c6a7dc8c0221371cf7d33a6ec16b8640d739728d23df5185f135b72f1e4b68e"}
Nov 03 16:19:00.232887 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:00.232359    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 16:19:00.232887 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:00.232420    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc" podUID="27399426-17f7-4ea7-aba6-12338b82c6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 16:19:00.390539 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:00.390459    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:01.123959 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.123663    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:01.238326 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.238293    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:19:01.239298 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.239278    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:19:01.240236 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.240213    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:19:01.369081 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.369010    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"
Nov 03 16:19:01.379729 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.376561    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"
Nov 03 16:19:01.464634 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.464602    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e94174aa-83ef-4037-861a-81dfbb6894d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94174aa-83ef-4037-861a-81dfbb6894d1-maas-api-openshift-ca\") pod \"e94174aa-83ef-4037-861a-81dfbb6894d1\" (UID: \"e94174aa-83ef-4037-861a-81dfbb6894d1\") "
Nov 03 16:19:01.465150 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.464656    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df53eb88-f023-4084-9e6a-eecbada27f7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df53eb88-f023-4084-9e6a-eecbada27f7a-maas-api-openshift-ca\") pod \"df53eb88-f023-4084-9e6a-eecbada27f7a\" (UID: \"df53eb88-f023-4084-9e6a-eecbada27f7a\") "
Nov 03 16:19:01.465150 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.464746    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e94174aa-83ef-4037-861a-81dfbb6894d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94174aa-83ef-4037-861a-81dfbb6894d1-maas-api-xks-ca\") pod \"e94174aa-83ef-4037-861a-81dfbb6894d1\" (UID: \"e94174aa-83ef-4037-861a-81dfbb6894d1\") "
Nov 03 16:19:01.465150 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.464776    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df53eb88-f023-4084-9e6a-eecbada27f7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df53eb88-f023-4084-9e6a-eecbada27f7a-maas-api-xks-ca\") pod \"df53eb88-f023-4084-9e6a-eecbada27f7a\" (UID: \"df53eb88-f023-4084-9e6a-eecbada27f7a\") "
Nov 03 16:19:01.465150 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.464836    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94174aa-83ef-4037-861a-81dfbb6894d1-kube-api-access-nt2fl\" (UniqueName: \"kubernetes.io/projected/e94174aa-83ef-4037-861a-81dfbb6894d1-kube-api-access-nt2fl\") pod \"e94174aa-83ef-4037-861a-81dfbb6894d1\" (UID: \"e94174aa-83ef-4037-861a-81dfbb6894d1\") "
Nov 03 16:19:01.465150 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.464855    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df53eb88-f023-4084-9e6a-eecbada27f7a-kube-api-access-8r2m5\" (UniqueName: \"kubernetes.io/projected/df53eb88-f023-4084-9e6a-eecbada27f7a-kube-api-access-8r2m5\") pod \"df53eb88-f023-4084-9e6a-eecbada27f7a\" (UID: \"df53eb88-f023-4084-9e6a-eecbada27f7a\") "
Nov 03 16:19:01.466232 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.466199    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94174aa-83ef-4037-861a-81dfbb6894d1-maas-api-openshift-ca" pod "e94174aa-83ef-4037-861a-81dfbb6894d1" (UID: "e94174aa-83ef-4037-861a-81dfbb6894d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:19:01.467811 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.466871    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df53eb88-f023-4084-9e6a-eecbada27f7a-maas-api-openshift-ca" pod "df53eb88-f023-4084-9e6a-eecbada27f7a" (UID: "df53eb88-f023-4084-9e6a-eecbada27f7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:19:01.468558 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.468531    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df53eb88-f023-4084-9e6a-eecbada27f7a-kube-api-access-8r2m5" pod "df53eb88-f023-4084-9e6a-eecbada27f7a" (UID: "df53eb88-f023-4084-9e6a-eecbada27f7a"). InnerVolumeSpecName "kube-api-access-8r2m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:19:01.468696 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.468674    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e94174aa-83ef-4037-861a-81dfbb6894d1-maas-api-xks-ca" pod "e94174aa-83ef-4037-861a-81dfbb6894d1" (UID: "e94174aa-83ef-4037-861a-81dfbb6894d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:19:01.469011 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.468988    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df53eb88-f023-4084-9e6a-eecbada27f7a-maas-api-xks-ca" pod "df53eb88-f023-4084-9e6a-eecbada27f7a" (UID: "df53eb88-f023-4084-9e6a-eecbada27f7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:19:01.469698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.469665    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94174aa-83ef-4037-861a-81dfbb6894d1-kube-api-access-nt2fl" pod "e94174aa-83ef-4037-861a-81dfbb6894d1" (UID: "e94174aa-83ef-4037-861a-81dfbb6894d1"). InnerVolumeSpecName "kube-api-access-nt2fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:19:01.565955 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.565915    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df53eb88-f023-4084-9e6a-eecbada27f7a-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:01.565955 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.565952    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94174aa-83ef-4037-861a-81dfbb6894d1-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:01.566162 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.565967    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df53eb88-f023-4084-9e6a-eecbada27f7a-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:01.566162 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.565979    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt2fl\" (UniqueName: \"kubernetes.io/projected/e94174aa-83ef-4037-861a-81dfbb6894d1-kube-api-access-nt2fl\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:01.566162 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.565991    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r2m5\" (UniqueName: \"kubernetes.io/projected/df53eb88-f023-4084-9e6a-eecbada27f7a-kube-api-access-8r2m5\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:01.566162 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:01.566004    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94174aa-83ef-4037-861a-81dfbb6894d1-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:02.457696 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:02.457655    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b" event={"ID":"df53eb88-f023-4084-9e6a-eecbada27f7a","Type":"ContainerDied","Data":"36aeb831dfbd0463755dad9bbc822d6e5ae4cd304fcbd5eb6eec644ea094af2b"}
Nov 03 16:19:02.458003 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:02.457983    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36aeb831dfbd0463755dad9bbc822d6e5ae4cd304fcbd5eb6eec644ea094af2b"
Nov 03 16:19:02.458203 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:02.458188    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"
Nov 03 16:19:02.494915 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:02.494881    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"
Nov 03 16:19:02.505644 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:02.505471    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:02.505975 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:02.505778    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9" event={"ID":"e94174aa-83ef-4037-861a-81dfbb6894d1","Type":"ContainerDied","Data":"1dec1a710b017b1f25c25e319c7c136942ec357379bc8b3e5d27c9d6b5334ba6"}
Nov 03 16:19:02.505975 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:02.505845    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dec1a710b017b1f25c25e319c7c136942ec357379bc8b3e5d27c9d6b5334ba6"
Nov 03 16:19:03.528780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:03.528748    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:04.495609 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:04.495558    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:04.876926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:04.876894    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:05.419738 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:05.419704    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:06.231437 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:06.231409    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:07.108740 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:07.108706    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:07.851436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:07.851394    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"]
Nov 03 16:19:07.852150 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:07.852130    6506 cpu_manager.go:414] "containerMap: removing container" podUID="e94174aa-83ef-4037-861a-81dfbb6894d1" containerName="revoke-keys"
Nov 03 16:19:07.852251 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:07.852152    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94174aa-83ef-4037-861a-81dfbb6894d1" containerName="revoke-keys"
Nov 03 16:19:07.852251 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:07.852184    6506 cpu_manager.go:414] "containerMap: removing container" podUID="df53eb88-f023-4084-9e6a-eecbada27f7a" containerName="revoke-keys"
Nov 03 16:19:07.852251 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:07.852192    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df53eb88-f023-4084-9e6a-eecbada27f7a" containerName="revoke-keys"
Nov 03 16:19:07.852395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:07.852337    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="df53eb88-f023-4084-9e6a-eecbada27f7a" containerName="revoke-keys"
Nov 03 16:19:07.852395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:07.852359    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94174aa-83ef-4037-861a-81dfbb6894d1" containerName="revoke-keys"
Nov 03 16:19:07.856955 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:07.856934    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"
Nov 03 16:19:07.863318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:07.863294    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:19:07.863499 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:07.863340    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:19:07.866988 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:07.866966    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"]
Nov 03 16:19:07.956305 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:07.956272    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkr5w\" (UniqueName: \"kubernetes.io/projected/fd81e299-8d66-4599-b914-54674a52d4cb-kube-api-access-vkr5w\") pod \"maas-api-revoke-sub-2051ba07-58d4t\" (UID: \"fd81e299-8d66-4599-b914-54674a52d4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"
Nov 03 16:19:07.956498 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:07.956426    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd81e299-8d66-4599-b914-54674a52d4cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2051ba07-58d4t\" (UID: \"fd81e299-8d66-4599-b914-54674a52d4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"
Nov 03 16:19:07.956590 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:07.956566    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd81e299-8d66-4599-b914-54674a52d4cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2051ba07-58d4t\" (UID: \"fd81e299-8d66-4599-b914-54674a52d4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"
Nov 03 16:19:08.057514 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:08.057479    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd81e299-8d66-4599-b914-54674a52d4cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2051ba07-58d4t\" (UID: \"fd81e299-8d66-4599-b914-54674a52d4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"
Nov 03 16:19:08.057687 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:08.057554    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd81e299-8d66-4599-b914-54674a52d4cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2051ba07-58d4t\" (UID: \"fd81e299-8d66-4599-b914-54674a52d4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"
Nov 03 16:19:08.057687 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:08.057673    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkr5w\" (UniqueName: \"kubernetes.io/projected/fd81e299-8d66-4599-b914-54674a52d4cb-kube-api-access-vkr5w\") pod \"maas-api-revoke-sub-2051ba07-58d4t\" (UID: \"fd81e299-8d66-4599-b914-54674a52d4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"
Nov 03 16:19:08.058181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:08.058159    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd81e299-8d66-4599-b914-54674a52d4cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2051ba07-58d4t\" (UID: \"fd81e299-8d66-4599-b914-54674a52d4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"
Nov 03 16:19:08.060098 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:08.060074    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd81e299-8d66-4599-b914-54674a52d4cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2051ba07-58d4t\" (UID: \"fd81e299-8d66-4599-b914-54674a52d4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"
Nov 03 16:19:08.076331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:08.076298    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkr5w\" (UniqueName: \"kubernetes.io/projected/fd81e299-8d66-4599-b914-54674a52d4cb-kube-api-access-vkr5w\") pod \"maas-api-revoke-sub-2051ba07-58d4t\" (UID: \"fd81e299-8d66-4599-b914-54674a52d4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"
Nov 03 16:19:08.172591 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:08.172518    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"
Nov 03 16:19:08.254698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:08.254393    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:08.465197 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:08.465174    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"]
Nov 03 16:19:08.466419 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:19:08.466394    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd81e299_8d66_4599_b914_54674a52d4cb.slice/crio-30e892d3d8496615acefbd3b98788aec590387d6e7f0338096530e93c5f49db8 WatchSource:0}: Error finding container 30e892d3d8496615acefbd3b98788aec590387d6e7f0338096530e93c5f49db8: Status 404 returned error can't find the container with id 30e892d3d8496615acefbd3b98788aec590387d6e7f0338096530e93c5f49db8
Nov 03 16:19:08.540500 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:08.540472    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t" event={"ID":"fd81e299-8d66-4599-b914-54674a52d4cb","Type":"ContainerStarted","Data":"30e892d3d8496615acefbd3b98788aec590387d6e7f0338096530e93c5f49db8"}
Nov 03 16:19:09.003616 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.003525    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:09.297120 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.297072    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:09.550724 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.550633    6506 generic.go:356] "Generic (PLEG): container finished" podID="fd81e299-8d66-4599-b914-54674a52d4cb" containerID="e222c19ccc00799672b33bbd83abfcb17301e661b4c4f39f220edc3722681a1f" exitCode=0
Nov 03 16:19:09.550926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.550719    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t" event={"ID":"fd81e299-8d66-4599-b914-54674a52d4cb","Type":"ContainerDied","Data":"e222c19ccc00799672b33bbd83abfcb17301e661b4c4f39f220edc3722681a1f"}
Nov 03 16:19:09.598669 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.598630    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"]
Nov 03 16:19:09.602515 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.602492    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.607075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.607052    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d569fb-openshift-default-dockercfg-wtrd5\""
Nov 03 16:19:09.620075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.620046    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"]
Nov 03 16:19:09.779947 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.779909    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05348976-46ce-43dc-9cd3-a7500709956a-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.780125 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.780019    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05348976-46ce-43dc-9cd3-a7500709956a-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.780125 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.780082    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9g6f\" (UniqueName: \"kubernetes.io/projected/05348976-46ce-43dc-9cd3-a7500709956a-kube-api-access-c9g6f\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.780264 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.780154    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05348976-46ce-43dc-9cd3-a7500709956a-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.780264 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.780182    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.780375 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.780262    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.780375 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.780288    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.780375 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.780309    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.780527 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.780465    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.882052 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.881975    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.882052 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.882011    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.882052 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.882031    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.882302 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.882096    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.882302 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.882136    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05348976-46ce-43dc-9cd3-a7500709956a-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.882302 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.882199    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05348976-46ce-43dc-9cd3-a7500709956a-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.882302 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.882232    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9g6f\" (UniqueName: \"kubernetes.io/projected/05348976-46ce-43dc-9cd3-a7500709956a-kube-api-access-c9g6f\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.882515 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.882302    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05348976-46ce-43dc-9cd3-a7500709956a-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.882515 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.882337    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.882515 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.882429    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.882515 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.882504    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.882715 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.882538    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.882715 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.882654    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.883040 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.883023    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05348976-46ce-43dc-9cd3-a7500709956a-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.884669 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.884650    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.884766 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.884750    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05348976-46ce-43dc-9cd3-a7500709956a-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.895513 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.895490    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05348976-46ce-43dc-9cd3-a7500709956a-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.895677 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.895660    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9g6f\" (UniqueName: \"kubernetes.io/projected/05348976-46ce-43dc-9cd3-a7500709956a-kube-api-access-c9g6f\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:09.917260 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:09.917236    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:10.035005 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.034827    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:10.210406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.210375    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"]
Nov 03 16:19:10.211884 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:19:10.211851    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05348976_46ce_43dc_9cd3_a7500709956a.slice/crio-80bd736887967badf044140730e40cc3d9db49418558bb9ee72b63e1ed44ab14 WatchSource:0}: Error finding container 80bd736887967badf044140730e40cc3d9db49418558bb9ee72b63e1ed44ab14: Status 404 returned error can't find the container with id 80bd736887967badf044140730e40cc3d9db49418558bb9ee72b63e1ed44ab14
Nov 03 16:19:10.214938 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.214896    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:19:10.215098 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.214973    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:19:10.215098 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.215003    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:19:10.559431 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.559387    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms" event={"ID":"05348976-46ce-43dc-9cd3-a7500709956a","Type":"ContainerStarted","Data":"6e41cb498685f4060b6e5015ff149d992633e310ad1bf2d35f444a1c0e4ab787"}
Nov 03 16:19:10.559595 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.559446    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms" event={"ID":"05348976-46ce-43dc-9cd3-a7500709956a","Type":"ContainerStarted","Data":"80bd736887967badf044140730e40cc3d9db49418558bb9ee72b63e1ed44ab14"}
Nov 03 16:19:10.596885 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.595288    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms" podStartSLOduration=1.595271135 podStartE2EDuration="1.595271135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:10.5908308 +0000 UTC m=+964.604228941" watchObservedRunningTime="2026-11-03 16:19:10.595271135 +0000 UTC m=+964.608669285"
Nov 03 16:19:10.747694 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.747668    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"
Nov 03 16:19:10.896028 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.895951    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd81e299-8d66-4599-b914-54674a52d4cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd81e299-8d66-4599-b914-54674a52d4cb-maas-api-openshift-ca\") pod \"fd81e299-8d66-4599-b914-54674a52d4cb\" (UID: \"fd81e299-8d66-4599-b914-54674a52d4cb\") "
Nov 03 16:19:10.896173 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.896072    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd81e299-8d66-4599-b914-54674a52d4cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd81e299-8d66-4599-b914-54674a52d4cb-maas-api-xks-ca\") pod \"fd81e299-8d66-4599-b914-54674a52d4cb\" (UID: \"fd81e299-8d66-4599-b914-54674a52d4cb\") "
Nov 03 16:19:10.896173 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.896113    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd81e299-8d66-4599-b914-54674a52d4cb-kube-api-access-vkr5w\" (UniqueName: \"kubernetes.io/projected/fd81e299-8d66-4599-b914-54674a52d4cb-kube-api-access-vkr5w\") pod \"fd81e299-8d66-4599-b914-54674a52d4cb\" (UID: \"fd81e299-8d66-4599-b914-54674a52d4cb\") "
Nov 03 16:19:10.896362 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.896336    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd81e299-8d66-4599-b914-54674a52d4cb-maas-api-openshift-ca" pod "fd81e299-8d66-4599-b914-54674a52d4cb" (UID: "fd81e299-8d66-4599-b914-54674a52d4cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:19:10.896524 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.896499    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd81e299-8d66-4599-b914-54674a52d4cb-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:10.898439 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.898413    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd81e299-8d66-4599-b914-54674a52d4cb-maas-api-xks-ca" pod "fd81e299-8d66-4599-b914-54674a52d4cb" (UID: "fd81e299-8d66-4599-b914-54674a52d4cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:19:10.898439 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.898424    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd81e299-8d66-4599-b914-54674a52d4cb-kube-api-access-vkr5w" pod "fd81e299-8d66-4599-b914-54674a52d4cb" (UID: "fd81e299-8d66-4599-b914-54674a52d4cb"). InnerVolumeSpecName "kube-api-access-vkr5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:19:10.928976 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.928941    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:10.997735 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.997704    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd81e299-8d66-4599-b914-54674a52d4cb-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:10.997735 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:10.997731    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkr5w\" (UniqueName: \"kubernetes.io/projected/fd81e299-8d66-4599-b914-54674a52d4cb-kube-api-access-vkr5w\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:11.131301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:11.124748    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:11.567118 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:11.567091    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"
Nov 03 16:19:11.567298 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:11.567132    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t" event={"ID":"fd81e299-8d66-4599-b914-54674a52d4cb","Type":"ContainerDied","Data":"30e892d3d8496615acefbd3b98788aec590387d6e7f0338096530e93c5f49db8"}
Nov 03 16:19:11.567298 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:11.567178    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30e892d3d8496615acefbd3b98788aec590387d6e7f0338096530e93c5f49db8"
Nov 03 16:19:11.933171 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:11.933079    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 16:19:11.933171 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:11.933144    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms" podUID="05348976-46ce-43dc-9cd3-a7500709956a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 16:19:12.206686 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:12.206596    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:12.930820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:12.929902    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 16:19:12.930820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:12.929991    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms" podUID="05348976-46ce-43dc-9cd3-a7500709956a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 16:19:13.274547 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:13.274515    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:13.931824 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:13.918142    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 16:19:13.931824 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:13.918204    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms" podUID="05348976-46ce-43dc-9cd3-a7500709956a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 16:19:13.946199 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:13.946163    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:14.270430 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:14.269719    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:14.932129 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:14.926114    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 16:19:14.932129 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:14.926174    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms" podUID="05348976-46ce-43dc-9cd3-a7500709956a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 16:19:15.196549 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:15.196460    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:15.591523 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:15.591479    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:15.935874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:15.935167    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 16:19:15.935874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:15.935224    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms" podUID="05348976-46ce-43dc-9cd3-a7500709956a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 16:19:16.451853 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:16.451812    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:16.925313 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:16.925281    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:16.925735 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:16.925710    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:16.926680 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:16.926658    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:19:17.445450 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:17.445416    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:18.891785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:18.891727    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:19.981078 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:19.981039    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:20.925209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:20.925171    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:21.864138 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:21.864090    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:22.724699 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:22.724657    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:23.508289 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.508250    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:23.772933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.772905    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"]
Nov 03 16:19:23.773466 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.773452    6506 cpu_manager.go:414] "containerMap: removing container" podUID="fd81e299-8d66-4599-b914-54674a52d4cb" containerName="revoke-keys"
Nov 03 16:19:23.773510 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.773467    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd81e299-8d66-4599-b914-54674a52d4cb" containerName="revoke-keys"
Nov 03 16:19:23.773555 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.773546    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd81e299-8d66-4599-b914-54674a52d4cb" containerName="revoke-keys"
Nov 03 16:19:23.777221 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.777194    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.796859 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.796829    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"]
Nov 03 16:19:23.885993 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.885957    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.885993 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.885995    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.886186 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.886128    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.886186 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.886168    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063fc995-de52-43cf-a70f-0f2d163e015f-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.886255 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.886197    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063fc995-de52-43cf-a70f-0f2d163e015f-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.886255 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.886214    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp7ps\" (UniqueName: \"kubernetes.io/projected/063fc995-de52-43cf-a70f-0f2d163e015f-kube-api-access-zp7ps\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.886321 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.886290    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.886354 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.886341    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.886431 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.886418    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063fc995-de52-43cf-a70f-0f2d163e015f-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.987434 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.987400    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063fc995-de52-43cf-a70f-0f2d163e015f-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.987588 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.987485    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.987588 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.987515    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.987588 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.987569    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.987726 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.987603    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063fc995-de52-43cf-a70f-0f2d163e015f-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.987726 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.987638    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063fc995-de52-43cf-a70f-0f2d163e015f-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.987726 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.987663    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp7ps\" (UniqueName: \"kubernetes.io/projected/063fc995-de52-43cf-a70f-0f2d163e015f-kube-api-access-zp7ps\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.987919 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.987822    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.987919 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.987913    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.988040 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.987976    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.988040 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.988012    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.988206 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.988182    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.988284 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.988266    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.988415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.988394    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063fc995-de52-43cf-a70f-0f2d163e015f-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.990200 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.990176    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:23.990286 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:23.990214    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063fc995-de52-43cf-a70f-0f2d163e015f-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:24.000505 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:24.000462    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp7ps\" (UniqueName: \"kubernetes.io/projected/063fc995-de52-43cf-a70f-0f2d163e015f-kube-api-access-zp7ps\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:24.000758 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:24.000738    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063fc995-de52-43cf-a70f-0f2d163e015f-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:24.100665 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:24.100591    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:24.412460 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:24.412359    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:24.421594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:24.421566    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"]
Nov 03 16:19:24.425133 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:19:24.425094    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063fc995_de52_43cf_a70f_0f2d163e015f.slice/crio-d891ed9597a3948b6972e8ba371d33ced9bc9776fa9c67616bc6d6d68037b413 WatchSource:0}: Error finding container d891ed9597a3948b6972e8ba371d33ced9bc9776fa9c67616bc6d6d68037b413: Status 404 returned error can't find the container with id d891ed9597a3948b6972e8ba371d33ced9bc9776fa9c67616bc6d6d68037b413
Nov 03 16:19:24.427749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:24.427717    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:19:24.427867 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:24.427818    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:19:24.427941 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:24.427865    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:19:24.944289 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:24.944241    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs" event={"ID":"063fc995-de52-43cf-a70f-0f2d163e015f","Type":"ContainerStarted","Data":"ccab854af11b6490a7106d30aba1e010efb884030e4e7add4c179586a2373f71"}
Nov 03 16:19:24.944765 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:24.944307    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs" event={"ID":"063fc995-de52-43cf-a70f-0f2d163e015f","Type":"ContainerStarted","Data":"d891ed9597a3948b6972e8ba371d33ced9bc9776fa9c67616bc6d6d68037b413"}
Nov 03 16:19:24.976933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:24.976887    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs" podStartSLOduration=1.976868595 podStartE2EDuration="1.976868595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:24.974846578 +0000 UTC m=+978.988244724" watchObservedRunningTime="2026-11-03 16:19:24.976868595 +0000 UTC m=+978.990266741"
Nov 03 16:19:25.101025 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:25.100986    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:25.102592 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:25.102565    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 16:19:25.102723 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:25.102616    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs" podUID="063fc995-de52-43cf-a70f-0f2d163e015f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 16:19:25.323545 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:25.323375    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:25.667865 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:25.667771    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:26.100998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:26.100965    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 16:19:26.101397 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:26.101021    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs" podUID="063fc995-de52-43cf-a70f-0f2d163e015f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 16:19:26.337485 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:26.337432    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:27.101388 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:27.101353    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 16:19:27.101774 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:27.101411    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs" podUID="063fc995-de52-43cf-a70f-0f2d163e015f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 16:19:27.161472 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:27.161434    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:28.101787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:28.101753    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 16:19:28.102279 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:28.101832    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs" podUID="063fc995-de52-43cf-a70f-0f2d163e015f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 16:19:28.138424 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:28.138386    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:28.797010 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:19:28.796972    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:19:29.028819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:29.028621    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:29.101143 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:29.101045    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 16:19:29.101143 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:29.101115    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs" podUID="063fc995-de52-43cf-a70f-0f2d163e015f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 16:19:29.929839 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:29.928078    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:30.101592 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:30.101562    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 16:19:30.101775 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:30.101611    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs" podUID="063fc995-de52-43cf-a70f-0f2d163e015f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 16:19:30.730556 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:30.730518    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:30.898924 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:30.898813    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:31.285634 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:31.285602    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:31.286413 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:31.286391    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:31.287236 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:31.287216    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:19:31.712489 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:31.712407    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:32.599978 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:32.599944    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:32.765702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:32.765674    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:33.437605 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:33.437566    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:34.217081 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:34.217035    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:35.104862 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:35.104782    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:36.900750 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:36.900716    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:38.141705 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:38.141661    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:38.884582 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:38.884544    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"]
Nov 03 16:19:38.897600 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:38.897565    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:38.918837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:38.918768    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"]
Nov 03 16:19:38.950519 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:38.950468    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:39.018348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.018313    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.018348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.018347    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqqh5\" (UniqueName: \"kubernetes.io/projected/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-kube-api-access-qqqh5\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.018596 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.018371    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.018596 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.018452    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.018728 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.018592    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.018728 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.018636    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.018728 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.018687    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.018878 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.018745    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.018878 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.018832    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.119648 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.119607    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.119648 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.119654    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.119910 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.119702    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.119910 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.119733    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.119910 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.119775    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.119910 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.119851    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.120126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.119984    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.120126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.120013    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqh5\" (UniqueName: \"kubernetes.io/projected/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-kube-api-access-qqqh5\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.120126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.120052    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.120126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.120080    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.120740 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.120321    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.120740 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.120376    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.120740 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.120471    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.120740 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.120506    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.122328 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.122296    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.122480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.122463    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.132210 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.132185    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqh5\" (UniqueName: \"kubernetes.io/projected/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-kube-api-access-qqqh5\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.132334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.132232    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.250419 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.250353    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:39.545583 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.545557    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"]
Nov 03 16:19:39.547937 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:19:39.547911    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f7e7a1_02a1_4c19_9749_ce5ddaa44dba.slice/crio-d0bac3761872a3ab7ebc04e9e764539e7a9052d150d3477d25b053a6f3fcb88c WatchSource:0}: Error finding container d0bac3761872a3ab7ebc04e9e764539e7a9052d150d3477d25b053a6f3fcb88c: Status 404 returned error can't find the container with id d0bac3761872a3ab7ebc04e9e764539e7a9052d150d3477d25b053a6f3fcb88c
Nov 03 16:19:39.550886 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.550841    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:19:39.550998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.550906    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:19:39.550998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.550936    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:19:39.911562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:39.911442    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:40.115168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:40.115136    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w" event={"ID":"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba","Type":"ContainerStarted","Data":"bae9d4a98f92a04a924f676f8f05dc81b0a314e180b77203f67cd2ed32dc6bbc"}
Nov 03 16:19:40.115348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:40.115184    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w" event={"ID":"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba","Type":"ContainerStarted","Data":"d0bac3761872a3ab7ebc04e9e764539e7a9052d150d3477d25b053a6f3fcb88c"}
Nov 03 16:19:40.142709 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:40.142655    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w" podStartSLOduration=2.14264218 podStartE2EDuration="2.14264218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:40.139593216 +0000 UTC m=+994.152991366" watchObservedRunningTime="2026-11-03 16:19:40.14264218 +0000 UTC m=+994.156040326"
Nov 03 16:19:40.252131 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:40.252000    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:40.254931 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:40.254356    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 16:19:40.254931 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:40.254436    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w" podUID="39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 16:19:40.693701 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:40.693628    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:41.251552 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:41.251516    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 16:19:41.251726 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:41.251568    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w" podUID="39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 16:19:41.502840 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:41.502709    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:42.251320 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:42.251283    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 16:19:42.251532 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:42.251342    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w" podUID="39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 16:19:42.416948 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:42.411452    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:43.251653 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:43.251613    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 16:19:43.252084 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:43.251670    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w" podUID="39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 16:19:43.299981 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:43.299940    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:43.948527 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:43.948496    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"]
Nov 03 16:19:43.952872 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:43.952848    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:43.961374 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:43.960859    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c09b17\"/\"kube-root-ca.crt\""
Nov 03 16:19:43.961374 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:43.961072    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c09b17\"/\"default-dockercfg-xbvrx\""
Nov 03 16:19:43.961374 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:43.961158    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c09b17\"/\"openshift-service-ca.crt\""
Nov 03 16:19:43.961374 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:43.961320    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c09b17\"/\"distinct-w1-c09b17-kserve-self-signed-certs\""
Nov 03 16:19:43.968265 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:43.968237    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"]
Nov 03 16:19:43.995211 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:43.995159    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:44.084320 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.084288    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-home\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.084482 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.084332    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-tmp-dir\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.084482 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.084411    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-model-cache\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.084482 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.084446    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjvmn\" (UniqueName: \"kubernetes.io/projected/65518370-5977-4201-9caa-029d28cf964d-kube-api-access-hjvmn\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.084482 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.084471    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-dshm\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.084638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.084615    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65518370-5977-4201-9caa-029d28cf964d-tls-certs\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.177821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.177681    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:44.185857 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.185828    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-home\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.186027 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.185881    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-tmp-dir\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.186027 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.185931    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-model-cache\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.186027 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.185972    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjvmn\" (UniqueName: \"kubernetes.io/projected/65518370-5977-4201-9caa-029d28cf964d-kube-api-access-hjvmn\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.186027 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.186007    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-dshm\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.186245 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.186177    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65518370-5977-4201-9caa-029d28cf964d-tls-certs\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.186315 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.186291    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-home\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.186999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.186975    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-tmp-dir\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.187225 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.187205    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-model-cache\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.189491 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.189468    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65518370-5977-4201-9caa-029d28cf964d-tls-certs\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.189593 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.189573    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-dshm\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.199745 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.199687    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjvmn\" (UniqueName: \"kubernetes.io/projected/65518370-5977-4201-9caa-029d28cf964d-kube-api-access-hjvmn\") pod \"distinct-w1-c09b17-kserve-6fc76765cb-dh8xv\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.251567 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.251522    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 16:19:44.251754 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.251587    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w" podUID="39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 16:19:44.273392 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.273366    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:44.578279 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.578252    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"]
Nov 03 16:19:44.580114 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:19:44.580086    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65518370_5977_4201_9caa_029d28cf964d.slice/crio-62153db20d98f19d1a8d4bb45b701f77ab4d771140452255c6d4e8a9aa360d70 WatchSource:0}: Error finding container 62153db20d98f19d1a8d4bb45b701f77ab4d771140452255c6d4e8a9aa360d70: Status 404 returned error can't find the container with id 62153db20d98f19d1a8d4bb45b701f77ab4d771140452255c6d4e8a9aa360d70
Nov 03 16:19:44.796648 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:44.796608    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:45.191358 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:45.191321    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" event={"ID":"65518370-5977-4201-9caa-029d28cf964d","Type":"ContainerStarted","Data":"c391304efb49bbf7c56702121140abef8afc97273b21c393454fca9a78e5b81d"}
Nov 03 16:19:45.191358 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:45.191363    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" event={"ID":"65518370-5977-4201-9caa-029d28cf964d","Type":"ContainerStarted","Data":"62153db20d98f19d1a8d4bb45b701f77ab4d771140452255c6d4e8a9aa360d70"}
Nov 03 16:19:45.221826 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:45.219286    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podStartSLOduration=2.219267279 podStartE2EDuration="2.219267279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:45.214132404 +0000 UTC m=+999.227530553" watchObservedRunningTime="2026-11-03 16:19:45.219267279 +0000 UTC m=+999.232665426"
Nov 03 16:19:45.251104 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:45.251074    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 16:19:45.251306 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:45.251125    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w" podUID="39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 16:19:45.273566 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:45.273542    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:45.622822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:45.619333    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:46.425821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:46.418533    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:46.501694 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:46.501502    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:46.502322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:46.502272    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:46.503584 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:46.503560    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:19:47.222054 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:47.221929    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:47.908662 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:47.908631    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:48.722900 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:48.722846    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:49.520551 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:49.520501    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:49.701211 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:49.701178    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:50.304232 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:50.304178    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:51.097371 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:51.097335    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:51.902476 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:51.902441    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:52.715328 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:52.715293    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:54.300869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.298576    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:54.326821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.323036    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:54.407027 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.406986    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"]
Nov 03 16:19:54.652595 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.652516    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.729825 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.719736    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"]
Nov 03 16:19:54.757780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.756893    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.757780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.756990    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.757780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.757114    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.757780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.757177    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/986e48d0-8cf9-45f3-86c7-06511006a2e1-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.757780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.757208    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.757780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.757253    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsp6l\" (UniqueName: \"kubernetes.io/projected/986e48d0-8cf9-45f3-86c7-06511006a2e1-kube-api-access-tsp6l\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.757780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.757345    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.757780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.757394    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.757780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.757531    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.768832 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.766543    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"]
Nov 03 16:19:54.773591 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.773572    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.815822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.811855    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"]
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.885175    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.888831    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.888881    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsp6l\" (UniqueName: \"kubernetes.io/projected/986e48d0-8cf9-45f3-86c7-06511006a2e1-kube-api-access-tsp6l\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.888955    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.888986    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889012    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z9gd\" (UniqueName: \"kubernetes.io/projected/5a76c7da-5ade-41c8-a075-430ccf3e7215-kube-api-access-2z9gd\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889056    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889108    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889168    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889197    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a76c7da-5ade-41c8-a075-430ccf3e7215-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889223    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889247    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889493    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889686    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889717    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889764    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889843    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889866    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889904    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889955    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/986e48d0-8cf9-45f3-86c7-06511006a2e1-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.889982    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.890587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.890464    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.892551 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.892525    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.893348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.893318    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/986e48d0-8cf9-45f3-86c7-06511006a2e1-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.893348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.893338    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.893643 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.893619    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.918557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.918491    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.932632 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.932574    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsp6l\" (UniqueName: \"kubernetes.io/projected/986e48d0-8cf9-45f3-86c7-06511006a2e1-kube-api-access-tsp6l\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:54.992334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.991195    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.992334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.991281    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.992334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.991359    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.992334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.991393    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9gd\" (UniqueName: \"kubernetes.io/projected/5a76c7da-5ade-41c8-a075-430ccf3e7215-kube-api-access-2z9gd\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.992334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.991421    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.992334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.991490    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.992334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.991516    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a76c7da-5ade-41c8-a075-430ccf3e7215-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.992334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.991563    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.992334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.991621    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.992334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.991657    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.992334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.992027    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.992334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.992077    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.992334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.992283    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.993123 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.992879    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a76c7da-5ade-41c8-a075-430ccf3e7215-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.994860 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.994834    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:54.995552 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:54.995518    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:55.005249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:55.005222    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:55.005384 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:55.005367    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9gd\" (UniqueName: \"kubernetes.io/projected/5a76c7da-5ade-41c8-a075-430ccf3e7215-kube-api-access-2z9gd\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:55.098963 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:55.098932    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:55.199047 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:55.198903    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:55.302548 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:55.302295    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:19:55.465992 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:55.465917    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"]
Nov 03 16:19:55.593053 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:55.591636    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:19:55.593053 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:55.591722    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:19:55.593053 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:55.591768    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:19:55.686854 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:55.681559    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:55.688296 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:55.688262    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"]
Nov 03 16:19:55.733318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:55.713548    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:19:55.733318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:55.713647    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:19:55.733318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:55.713721    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:19:56.056883 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:56.056787    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:56.374317 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:56.374212    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2" event={"ID":"5a76c7da-5ade-41c8-a075-430ccf3e7215","Type":"ContainerStarted","Data":"90b4c690b6dd52b9577f008d24d2b39d7c076fa2fd75e0a421c73138c6cbbe4c"}
Nov 03 16:19:56.374317 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:56.374282    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2" event={"ID":"5a76c7da-5ade-41c8-a075-430ccf3e7215","Type":"ContainerStarted","Data":"2138513c6571110321c2eab24307d0b16b7d9e29b4af2ea2db9f709131533b72"}
Nov 03 16:19:56.376152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:56.376121    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj" event={"ID":"986e48d0-8cf9-45f3-86c7-06511006a2e1","Type":"ContainerStarted","Data":"28e7f6ac50570950bd5ce77ffaada26044e4ce5d4cddea9c89db7d8fc0127025"}
Nov 03 16:19:56.376270 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:56.376178    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj" event={"ID":"986e48d0-8cf9-45f3-86c7-06511006a2e1","Type":"ContainerStarted","Data":"0ef9aae596b7d06fdd9f2e58a602ea44f497ee5c4fee0028c0c21a01800ec166"}
Nov 03 16:19:56.403330 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:56.403273    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2" podStartSLOduration=2.40325578 podStartE2EDuration="2.40325578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:56.400878005 +0000 UTC m=+1010.414276152" watchObservedRunningTime="2026-11-03 16:19:56.40325578 +0000 UTC m=+1010.416653927"
Nov 03 16:19:56.429958 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:56.429906    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj" podStartSLOduration=2.429891674 podStartE2EDuration="2.429891674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:56.427779851 +0000 UTC m=+1010.441177999" watchObservedRunningTime="2026-11-03 16:19:56.429891674 +0000 UTC m=+1010.443289820"
Nov 03 16:19:56.699312 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:56.699217    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:57.099295 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:57.099257    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:19:57.101020 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:57.100990    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 16:19:57.101132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:57.101050    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj" podUID="986e48d0-8cf9-45f3-86c7-06511006a2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 16:19:57.199332 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:57.199295    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:19:57.200692 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:57.200665    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 16:19:57.200840 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:57.200714    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2" podUID="5a76c7da-5ade-41c8-a075-430ccf3e7215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 16:19:57.542821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:57.529457    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:58.099405 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:58.099369    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 16:19:58.099560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:58.099422    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj" podUID="986e48d0-8cf9-45f3-86c7-06511006a2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 16:19:58.199954 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:58.199920    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 16:19:58.200127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:58.199980    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2" podUID="5a76c7da-5ade-41c8-a075-430ccf3e7215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 16:19:58.254820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:58.254621    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:58.798779 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:19:58.798742    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:19:59.054367 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:59.054240    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:59.099443 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:59.099413    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 16:19:59.099608 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:59.099471    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj" podUID="986e48d0-8cf9-45f3-86c7-06511006a2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 16:19:59.203353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:59.203306    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 16:19:59.203635 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:59.203588    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2" podUID="5a76c7da-5ade-41c8-a075-430ccf3e7215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 16:19:59.252877 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:59.252844    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:19:59.842351 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:19:59.842311    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:00.099657 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:00.099558    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 16:20:00.099657 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:00.099602    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj" podUID="986e48d0-8cf9-45f3-86c7-06511006a2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 16:20:00.199822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:00.199766    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 16:20:00.199976 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:00.199834    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2" podUID="5a76c7da-5ade-41c8-a075-430ccf3e7215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 16:20:00.762821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:00.762662    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:01.064822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.063165    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"]
Nov 03 16:20:01.068450 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.068425    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.073919 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.073846    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 16:20:01.074647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.074574    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 16:20:01.075137 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.074862    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 16:20:01.075137 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.075051    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6779w\""
Nov 03 16:20:01.080539 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.080518    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"]
Nov 03 16:20:01.098732 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.098704    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.098944 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.098929    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.099134 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.099111    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.099264 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.099245    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab567ff1-957a-4316-9939-f0977aeecde7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.099338 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.099322    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab567ff1-957a-4316-9939-f0977aeecde7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.099388 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.099371    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvqz5\" (UniqueName: \"kubernetes.io/projected/ab567ff1-957a-4316-9939-f0977aeecde7-kube-api-access-fvqz5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.099438 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.099425    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab567ff1-957a-4316-9939-f0977aeecde7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.099631 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.099610    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 16:20:01.099691 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.099668    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj" podUID="986e48d0-8cf9-45f3-86c7-06511006a2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 16:20:01.199965 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.199931    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 16:20:01.200121 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.199982    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2" podUID="5a76c7da-5ade-41c8-a075-430ccf3e7215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 16:20:01.200881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.200862    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.200952 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.200938    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab567ff1-957a-4316-9939-f0977aeecde7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.200988 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.200969    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab567ff1-957a-4316-9939-f0977aeecde7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.201033 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.200994    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvqz5\" (UniqueName: \"kubernetes.io/projected/ab567ff1-957a-4316-9939-f0977aeecde7-kube-api-access-fvqz5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.201033 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.201023    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab567ff1-957a-4316-9939-f0977aeecde7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.201105 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.201042    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.201105 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.201071    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.202419 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.202360    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab567ff1-957a-4316-9939-f0977aeecde7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.203957 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.203905    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.204173 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.204154    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.204404 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.204388    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.204741 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.204723    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab567ff1-957a-4316-9939-f0977aeecde7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.257585 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.257547    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvqz5\" (UniqueName: \"kubernetes.io/projected/ab567ff1-957a-4316-9939-f0977aeecde7-kube-api-access-fvqz5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.266422 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.266358    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab567ff1-957a-4316-9939-f0977aeecde7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trdp7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.385761 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.385684    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:01.584821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.553620    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:01.788283 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:01.788256    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"]
Nov 03 16:20:01.788625 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:20:01.788603    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab567ff1_957a_4316_9939_f0977aeecde7.slice/crio-8e09d8c164b0a04e9559091287f3702c6ec79b1e01329c7d3bd768ec2118868b WatchSource:0}: Error finding container 8e09d8c164b0a04e9559091287f3702c6ec79b1e01329c7d3bd768ec2118868b: Status 404 returned error can't find the container with id 8e09d8c164b0a04e9559091287f3702c6ec79b1e01329c7d3bd768ec2118868b
Nov 03 16:20:02.100292 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:02.100214    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 16:20:02.100292 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:02.100263    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj" podUID="986e48d0-8cf9-45f3-86c7-06511006a2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 16:20:02.200209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:02.200176    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 16:20:02.200383 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:02.200233    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2" podUID="5a76c7da-5ade-41c8-a075-430ccf3e7215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 16:20:02.323844 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:02.322052    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:02.480693 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:02.480595    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7" event={"ID":"ab567ff1-957a-4316-9939-f0977aeecde7","Type":"ContainerStarted","Data":"8e09d8c164b0a04e9559091287f3702c6ec79b1e01329c7d3bd768ec2118868b"}
Nov 03 16:20:03.274821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:03.268990    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:03.597192 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:03.596135    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:20:03.597192 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:03.596978    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:20:03.599184 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:03.599138    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:20:03.711617 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:03.709385    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:20:03.711617 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:03.710927    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:20:03.712591 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:03.712556    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:20:04.097833 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:04.097764    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:04.791275 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:04.791232    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:05.231919 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:05.231887    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:05.232043 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:05.231953    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:05.519669 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:05.519627    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7" event={"ID":"ab567ff1-957a-4316-9939-f0977aeecde7","Type":"ContainerStarted","Data":"4c66698a2713b66534c4ef978011b746a08499f03bbcf22551930831cf29800a"}
Nov 03 16:20:05.520130 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:05.520082    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:05.550774 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:05.550706    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7" podStartSLOduration=1.110664726 podStartE2EDuration="4.55069017s" totalImagesPullingTime="3.440025444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:05.550014306 +0000 UTC m=+1019.563412451" watchObservedRunningTime="2026-11-03 16:20:05.55069017 +0000 UTC m=+1019.564088323"
Nov 03 16:20:05.600408 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:05.600368    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:06.401880 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:06.401834    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:06.530130 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:06.530100    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:20:07.129666 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:07.129627    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:07.811406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:07.811334    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:08.582600 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:08.582538    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:08.745119 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:08.745079    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:09.350535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.350497    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:09.743610 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.743532    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"]
Nov 03 16:20:09.748904 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.748886    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.766061 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.766036    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"]
Nov 03 16:20:09.781960 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.781931    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.782090 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.781964    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.782150 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.782127    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.782204 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.782161    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.782204 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.782194    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkgn6\" (UniqueName: \"kubernetes.io/projected/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-kube-api-access-vkgn6\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.782314 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.782291    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.782373 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.782357    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.782422 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.782386    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.782474 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.782442    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.800233 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.800205    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:09.883349 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.883313    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkgn6\" (UniqueName: \"kubernetes.io/projected/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-kube-api-access-vkgn6\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.883349 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.883353    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.883577 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.883409    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.883577 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.883427    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.883577 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.883460    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.883577 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.883523    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.883577 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.883538    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.883736 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.883649    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.883736 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.883684    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.884075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.884039    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.884785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.884762    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.885009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.884993    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.885208 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.885176    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.885363 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.885219    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.887306 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.887284    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.887878 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.887854    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.896470 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.896446    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkgn6\" (UniqueName: \"kubernetes.io/projected/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-kube-api-access-vkgn6\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:09.896554 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:09.896501    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:10.071849 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:10.070257    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:10.385245 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:10.385171    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"]
Nov 03 16:20:10.394690 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:20:10.394657    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9066cf9a_5d56_4a7d_b60b_cc881a2a0138.slice/crio-9da26764a91d5dc202a8641323b68c7962f65ff0a36fb99c32ecdeef140423c9 WatchSource:0}: Error finding container 9da26764a91d5dc202a8641323b68c7962f65ff0a36fb99c32ecdeef140423c9: Status 404 returned error can't find the container with id 9da26764a91d5dc202a8641323b68c7962f65ff0a36fb99c32ecdeef140423c9
Nov 03 16:20:10.397157 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:10.397126    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:10.397273 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:10.397193    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:10.397273 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:10.397221    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:10.567939 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:10.567907    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx" event={"ID":"9066cf9a-5d56-4a7d-b60b-cc881a2a0138","Type":"ContainerStarted","Data":"d5b16f7aa6a4d29f81f88813f5a3012ee9a326af227e152d011963332aebefde"}
Nov 03 16:20:10.568123 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:10.567964    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx" event={"ID":"9066cf9a-5d56-4a7d-b60b-cc881a2a0138","Type":"ContainerStarted","Data":"9da26764a91d5dc202a8641323b68c7962f65ff0a36fb99c32ecdeef140423c9"}
Nov 03 16:20:10.595406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:10.595353    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx" podStartSLOduration=1.5953361689999999 podStartE2EDuration="1.595336169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:10.591740158 +0000 UTC m=+1024.605138306" watchObservedRunningTime="2026-11-03 16:20:10.595336169 +0000 UTC m=+1024.608734317"
Nov 03 16:20:10.819596 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:10.819562    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:11.070915 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:11.070843    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:11.072085 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:11.072061    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 16:20:11.072197 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:11.072117    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx" podUID="9066cf9a-5d56-4a7d-b60b-cc881a2a0138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 16:20:11.832626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:11.832260    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:12.071818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:12.071767    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 16:20:12.071987 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:12.071865    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx" podUID="9066cf9a-5d56-4a7d-b60b-cc881a2a0138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 16:20:12.855666 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:12.855603    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:13.071369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:13.071330    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 16:20:13.071531 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:13.071392    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx" podUID="9066cf9a-5d56-4a7d-b60b-cc881a2a0138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 16:20:13.890677 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:13.890619    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:14.070907 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:14.070873    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 16:20:14.071085 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:14.070932    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx" podUID="9066cf9a-5d56-4a7d-b60b-cc881a2a0138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 16:20:15.078820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:15.076062    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 16:20:15.078820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:15.076126    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx" podUID="9066cf9a-5d56-4a7d-b60b-cc881a2a0138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 16:20:15.418890 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:15.416326    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:16.104933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:16.073289    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 16:20:16.119896 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:16.119853    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx" podUID="9066cf9a-5d56-4a7d-b60b-cc881a2a0138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 16:20:16.408210 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:16.329243    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:16.597333 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:16.597288    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:17.014981 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:17.014926    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:17.072384 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:17.072346    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 16:20:17.072536 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:17.072429    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx" podUID="9066cf9a-5d56-4a7d-b60b-cc881a2a0138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 16:20:17.890012 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:17.889977    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:18.305435 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:18.305406    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:18.305883 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:18.305860    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:18.306997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:18.306845    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:20:18.810666 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:18.810638    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:19.074622 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:19.074535    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:19.711044 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:19.711007    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:20.811617 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:20.811583    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:21.610966 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:21.610927    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:22.614602 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:22.614557    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:23.209820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:23.209759    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:23.635089 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:23.635030    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:24.612839 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:24.611652    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:25.539452 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:25.539416    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:26.342219 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:26.342191    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:27.110840 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:27.110786    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:27.916258 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:27.916226    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:28.798152 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:20:28.798111    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:20:28.935301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:28.935262    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:29.805963 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:29.805927    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:30.802526 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:30.802473    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:31.901396 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:31.901364    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:32.616463 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:32.616427    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:33.492455 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:33.492414    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:34.320279 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:34.320238    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:35.202033 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:35.201990    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:36.355898 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:36.322381    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:36.835291 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:36.835219    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"]
Nov 03 16:20:36.835815 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:36.835767    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr" podUID="bb1ab2e2-737e-490f-b820-8ce3be90c1b5" containerName="istio-proxy" containerID="cri-o://84cb5cddef36a6812b2953b7ab30ca5587fa6b40a1f323fb460f74c51157bf30" gracePeriod=30
Nov 03 16:20:36.921625 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:36.921589    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:38.951505 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:38.951470    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:39.151279 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.151242    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"]
Nov 03 16:20:39.161112 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.161073    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.173073 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.173007    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"]
Nov 03 16:20:39.255577 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.255494    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.255577 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.255542    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.255788 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.255659    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.255788 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.255686    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.255944 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.255901    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.256028 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.256014    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.256113 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.256098    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.256190 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.256167    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.256287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.256266    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dk55\" (UniqueName: \"kubernetes.io/projected/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-kube-api-access-4dk55\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.357886 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.357845    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.358035 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.357915    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dk55\" (UniqueName: \"kubernetes.io/projected/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-kube-api-access-4dk55\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.358035 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.357992    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.358035 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.358016    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.358207 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.358090    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.358207 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.358109    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.358207 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.358132    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.358207 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.358171    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.358419 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.358235    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.358635 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.358605    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.358757 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.358654    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.358921 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.358870    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.358921 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.358879    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.358921 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.358885    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.360755 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.360727    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.361349 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.361328    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.370902 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.370876    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.370992 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.370937    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dk55\" (UniqueName: \"kubernetes.io/projected/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-kube-api-access-4dk55\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.483924 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.483893    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:39.790927 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.790900    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"]
Nov 03 16:20:39.793901 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:20:39.793869    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9d0efd_35df_4e0e_a9fb_dff1b6ca36c1.slice/crio-e2db75b9b1bee08b228cdabde8b969e6c0c9cf997c74660dc9bd458810c79c6c WatchSource:0}: Error finding container e2db75b9b1bee08b228cdabde8b969e6c0c9cf997c74660dc9bd458810c79c6c: Status 404 returned error can't find the container with id e2db75b9b1bee08b228cdabde8b969e6c0c9cf997c74660dc9bd458810c79c6c
Nov 03 16:20:39.796641 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.796576    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:39.796706 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.796646    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:39.796706 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.796674    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:39.959650 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.959607    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894" event={"ID":"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1","Type":"ContainerStarted","Data":"a6c706849899f6371030aa95b8e314fe6d9094638ed61d2543d8785697757d45"}
Nov 03 16:20:39.960157 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.959674    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894" event={"ID":"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1","Type":"ContainerStarted","Data":"e2db75b9b1bee08b228cdabde8b969e6c0c9cf997c74660dc9bd458810c79c6c"}
Nov 03 16:20:39.988401 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:39.988367    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:40.011552 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:40.006412    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894" podStartSLOduration=1.006389287 podStartE2EDuration="1.006389287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:39.991332541 +0000 UTC m=+1054.004730711" watchObservedRunningTime="2026-11-03 16:20:40.006389287 +0000 UTC m=+1054.019787433"
Nov 03 16:20:40.486517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:40.486479    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:40.490503 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:40.490459    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 16:20:40.490684 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:40.490519    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894" podUID="de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 16:20:41.063374 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:41.063341    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:41.484350 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:41.484272    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 16:20:41.484350 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:41.484325    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894" podUID="de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 16:20:41.974852 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:41.974825    6506 generic.go:356] "Generic (PLEG): container finished" podID="bb1ab2e2-737e-490f-b820-8ce3be90c1b5" containerID="84cb5cddef36a6812b2953b7ab30ca5587fa6b40a1f323fb460f74c51157bf30" exitCode=0
Nov 03 16:20:41.974966 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:41.974894    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr" event={"ID":"bb1ab2e2-737e-490f-b820-8ce3be90c1b5","Type":"ContainerDied","Data":"84cb5cddef36a6812b2953b7ab30ca5587fa6b40a1f323fb460f74c51157bf30"}
Nov 03 16:20:42.092775 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.092743    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:42.192383 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.192361    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:20:42.267689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.267656    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-podinfo\") pod \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") "
Nov 03 16:20:42.267888 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.267756    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-workload-certs\") pod \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") "
Nov 03 16:20:42.267888 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.267838    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-envoy\") pod \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") "
Nov 03 16:20:42.268019 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.267904    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-kube-api-access-drg4n\" (UniqueName: \"kubernetes.io/projected/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-kube-api-access-drg4n\") pod \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") "
Nov 03 16:20:42.268019 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.267966    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-token\" (UniqueName: \"kubernetes.io/projected/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-token\") pod \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") "
Nov 03 16:20:42.268124 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.268017    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-credential-socket\") pod \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") "
Nov 03 16:20:42.268124 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.268073    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-workload-socket\") pod \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") "
Nov 03 16:20:42.268124 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.268113    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istiod-ca-cert\") pod \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") "
Nov 03 16:20:42.268281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.268142    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-data\") pod \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\" (UID: \"bb1ab2e2-737e-490f-b820-8ce3be90c1b5\") "
Nov 03 16:20:42.269426 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.269399    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-workload-certs" pod "bb1ab2e2-737e-490f-b820-8ce3be90c1b5" (UID: "bb1ab2e2-737e-490f-b820-8ce3be90c1b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:20:42.269592 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.269413    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-workload-socket" pod "bb1ab2e2-737e-490f-b820-8ce3be90c1b5" (UID: "bb1ab2e2-737e-490f-b820-8ce3be90c1b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:20:42.269698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.269625    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-credential-socket" pod "bb1ab2e2-737e-490f-b820-8ce3be90c1b5" (UID: "bb1ab2e2-737e-490f-b820-8ce3be90c1b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:20:42.272824 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.269947    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istiod-ca-cert" pod "bb1ab2e2-737e-490f-b820-8ce3be90c1b5" (UID: "bb1ab2e2-737e-490f-b820-8ce3be90c1b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:20:42.272824 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.270042    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-data" pod "bb1ab2e2-737e-490f-b820-8ce3be90c1b5" (UID: "bb1ab2e2-737e-490f-b820-8ce3be90c1b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:20:42.272824 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.271513    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-podinfo" pod "bb1ab2e2-737e-490f-b820-8ce3be90c1b5" (UID: "bb1ab2e2-737e-490f-b820-8ce3be90c1b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:20:42.275451 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.275384    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-token" pod "bb1ab2e2-737e-490f-b820-8ce3be90c1b5" (UID: "bb1ab2e2-737e-490f-b820-8ce3be90c1b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:20:42.275938 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.275910    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-envoy" pod "bb1ab2e2-737e-490f-b820-8ce3be90c1b5" (UID: "bb1ab2e2-737e-490f-b820-8ce3be90c1b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:20:42.278367 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.278332    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-kube-api-access-drg4n" pod "bb1ab2e2-737e-490f-b820-8ce3be90c1b5" (UID: "bb1ab2e2-737e-490f-b820-8ce3be90c1b5"). InnerVolumeSpecName "kube-api-access-drg4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:20:42.369646 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.369616    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:42.369646 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.369642    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:42.369646 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.369652    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drg4n\" (UniqueName: \"kubernetes.io/projected/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-kube-api-access-drg4n\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:42.369903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.369663    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:42.369903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.369672    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:42.369903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.369680    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:42.369903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.369689    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:42.369903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.369698    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:42.369903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.369706    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1ab2e2-737e-490f-b820-8ce3be90c1b5-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:42.484280 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.484201    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 16:20:42.484280 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.484251    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894" podUID="de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 16:20:42.982780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.982753    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"
Nov 03 16:20:42.982957 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.982753    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr" event={"ID":"bb1ab2e2-737e-490f-b820-8ce3be90c1b5","Type":"ContainerDied","Data":"764e57f78891c745d2d47b8218970c5f207773013c5db2b244ac79b2059c59e0"}
Nov 03 16:20:42.982957 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:42.982890    6506 scope.go:122] "RemoveContainer" containerID="84cb5cddef36a6812b2953b7ab30ca5587fa6b40a1f323fb460f74c51157bf30"
Nov 03 16:20:43.029827 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:43.026859    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"]
Nov 03 16:20:43.034033 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:43.034006    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bs7fr"]
Nov 03 16:20:43.244878 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:43.234951    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:43.484958 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:43.484926    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 16:20:43.485133 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:43.484978    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894" podUID="de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 16:20:44.104833 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.104779    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb1ab2e2-737e-490f-b820-8ce3be90c1b5" path="/var/lib/kubelet/pods/bb1ab2e2-737e-490f-b820-8ce3be90c1b5/volumes"
Nov 03 16:20:44.463751 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.463660    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"]
Nov 03 16:20:44.464597 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.464566    6506 cpu_manager.go:414] "containerMap: removing container" podUID="bb1ab2e2-737e-490f-b820-8ce3be90c1b5" containerName="istio-proxy"
Nov 03 16:20:44.464597 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.464591    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb1ab2e2-737e-490f-b820-8ce3be90c1b5" containerName="istio-proxy"
Nov 03 16:20:44.464854 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.464772    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb1ab2e2-737e-490f-b820-8ce3be90c1b5" containerName="istio-proxy"
Nov 03 16:20:44.470337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.470313    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"
Nov 03 16:20:44.477319 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.477293    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:20:44.478709 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.478669    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:20:44.483146 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.482359    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:44.490175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.484939    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 16:20:44.490175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.484987    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894" podUID="de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 16:20:44.496097 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.496075    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"]
Nov 03 16:20:44.497997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.497973    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s4sz\" (UniqueName: \"kubernetes.io/projected/ca5f4ac5-141c-4b00-869d-942cc8198b1d-kube-api-access-5s4sz\") pod \"maas-api-revoke-sub-b7503e5f-lzvjk\" (UID: \"ca5f4ac5-141c-4b00-869d-942cc8198b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"
Nov 03 16:20:44.498153 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.498135    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5f4ac5-141c-4b00-869d-942cc8198b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7503e5f-lzvjk\" (UID: \"ca5f4ac5-141c-4b00-869d-942cc8198b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"
Nov 03 16:20:44.498209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.498169    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5f4ac5-141c-4b00-869d-942cc8198b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7503e5f-lzvjk\" (UID: \"ca5f4ac5-141c-4b00-869d-942cc8198b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"
Nov 03 16:20:44.599994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.599959    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5f4ac5-141c-4b00-869d-942cc8198b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7503e5f-lzvjk\" (UID: \"ca5f4ac5-141c-4b00-869d-942cc8198b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"
Nov 03 16:20:44.600183 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.600009    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5f4ac5-141c-4b00-869d-942cc8198b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7503e5f-lzvjk\" (UID: \"ca5f4ac5-141c-4b00-869d-942cc8198b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"
Nov 03 16:20:44.600183 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.600174    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s4sz\" (UniqueName: \"kubernetes.io/projected/ca5f4ac5-141c-4b00-869d-942cc8198b1d-kube-api-access-5s4sz\") pod \"maas-api-revoke-sub-b7503e5f-lzvjk\" (UID: \"ca5f4ac5-141c-4b00-869d-942cc8198b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"
Nov 03 16:20:44.600605 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.600579    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5f4ac5-141c-4b00-869d-942cc8198b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7503e5f-lzvjk\" (UID: \"ca5f4ac5-141c-4b00-869d-942cc8198b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"
Nov 03 16:20:44.602647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.602622    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5f4ac5-141c-4b00-869d-942cc8198b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7503e5f-lzvjk\" (UID: \"ca5f4ac5-141c-4b00-869d-942cc8198b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"
Nov 03 16:20:44.633773 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.633729    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s4sz\" (UniqueName: \"kubernetes.io/projected/ca5f4ac5-141c-4b00-869d-942cc8198b1d-kube-api-access-5s4sz\") pod \"maas-api-revoke-sub-b7503e5f-lzvjk\" (UID: \"ca5f4ac5-141c-4b00-869d-942cc8198b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"
Nov 03 16:20:44.790941 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:44.790915    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"
Nov 03 16:20:45.146344 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:45.146312    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"]
Nov 03 16:20:45.484851 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:45.484761    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 16:20:45.485210 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:45.484841    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894" podUID="de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 16:20:45.983509 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:45.983433    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:46.006475 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:46.006436    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk" event={"ID":"ca5f4ac5-141c-4b00-869d-942cc8198b1d","Type":"ContainerStarted","Data":"8c49a6b6d0b5142f6fc1d722687037b5db8c3b2558f10a1f5f336925d3c11cb5"}
Nov 03 16:20:46.006654 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:46.006494    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk" event={"ID":"ca5f4ac5-141c-4b00-869d-942cc8198b1d","Type":"ContainerStarted","Data":"8c13cbdcebeacfb6074c53fa8dbae2a812593364747deb87506c33ab9d311b1f"}
Nov 03 16:20:46.029938 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:46.029888    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk" podStartSLOduration=2.02987031 podStartE2EDuration="2.02987031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:46.028978154 +0000 UTC m=+1060.042376313" watchObservedRunningTime="2026-11-03 16:20:46.02987031 +0000 UTC m=+1060.043268456"
Nov 03 16:20:46.485152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:46.485119    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 16:20:46.485553 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:46.485172    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894" podUID="de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 16:20:47.001395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:47.001359    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:47.018398 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:47.018361    6506 generic.go:356] "Generic (PLEG): container finished" podID="ca5f4ac5-141c-4b00-869d-942cc8198b1d" containerID="8c49a6b6d0b5142f6fc1d722687037b5db8c3b2558f10a1f5f336925d3c11cb5" exitCode=0
Nov 03 16:20:47.018557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:47.018455    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk" event={"ID":"ca5f4ac5-141c-4b00-869d-942cc8198b1d","Type":"ContainerDied","Data":"8c49a6b6d0b5142f6fc1d722687037b5db8c3b2558f10a1f5f336925d3c11cb5"}
Nov 03 16:20:47.488836 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:47.488807    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:47.489324 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:47.489303    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:47.490021 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:47.490002    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:20:48.070618 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:48.070585    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:48.210724 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:48.210699    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"
Nov 03 16:20:48.296661 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:48.296582    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5f4ac5-141c-4b00-869d-942cc8198b1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5f4ac5-141c-4b00-869d-942cc8198b1d-maas-api-openshift-ca\") pod \"ca5f4ac5-141c-4b00-869d-942cc8198b1d\" (UID: \"ca5f4ac5-141c-4b00-869d-942cc8198b1d\") "
Nov 03 16:20:48.296859 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:48.296745    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5f4ac5-141c-4b00-869d-942cc8198b1d-kube-api-access-5s4sz\" (UniqueName: \"kubernetes.io/projected/ca5f4ac5-141c-4b00-869d-942cc8198b1d-kube-api-access-5s4sz\") pod \"ca5f4ac5-141c-4b00-869d-942cc8198b1d\" (UID: \"ca5f4ac5-141c-4b00-869d-942cc8198b1d\") "
Nov 03 16:20:48.296940 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:48.296854    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca5f4ac5-141c-4b00-869d-942cc8198b1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5f4ac5-141c-4b00-869d-942cc8198b1d-maas-api-xks-ca\") pod \"ca5f4ac5-141c-4b00-869d-942cc8198b1d\" (UID: \"ca5f4ac5-141c-4b00-869d-942cc8198b1d\") "
Nov 03 16:20:48.297030 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:48.296999    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5f4ac5-141c-4b00-869d-942cc8198b1d-maas-api-openshift-ca" pod "ca5f4ac5-141c-4b00-869d-942cc8198b1d" (UID: "ca5f4ac5-141c-4b00-869d-942cc8198b1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:20:48.297459 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:48.297439    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5f4ac5-141c-4b00-869d-942cc8198b1d-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:48.298963 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:48.298933    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5f4ac5-141c-4b00-869d-942cc8198b1d-maas-api-xks-ca" pod "ca5f4ac5-141c-4b00-869d-942cc8198b1d" (UID: "ca5f4ac5-141c-4b00-869d-942cc8198b1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:20:48.299093 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:48.299068    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5f4ac5-141c-4b00-869d-942cc8198b1d-kube-api-access-5s4sz" pod "ca5f4ac5-141c-4b00-869d-942cc8198b1d" (UID: "ca5f4ac5-141c-4b00-869d-942cc8198b1d"). InnerVolumeSpecName "kube-api-access-5s4sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:20:48.399594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:48.399565    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5f4ac5-141c-4b00-869d-942cc8198b1d-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:48.399594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:48.399593    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s4sz\" (UniqueName: \"kubernetes.io/projected/ca5f4ac5-141c-4b00-869d-942cc8198b1d-kube-api-access-5s4sz\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:49.044030 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.043995    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk" event={"ID":"ca5f4ac5-141c-4b00-869d-942cc8198b1d","Type":"ContainerDied","Data":"8c13cbdcebeacfb6074c53fa8dbae2a812593364747deb87506c33ab9d311b1f"}
Nov 03 16:20:49.044508 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.044056    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c13cbdcebeacfb6074c53fa8dbae2a812593364747deb87506c33ab9d311b1f"
Nov 03 16:20:49.044508 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.044013    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"
Nov 03 16:20:49.089399 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.089365    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:49.374748 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.374672    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"]
Nov 03 16:20:49.375283 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.375270    6506 cpu_manager.go:414] "containerMap: removing container" podUID="ca5f4ac5-141c-4b00-869d-942cc8198b1d" containerName="revoke-keys"
Nov 03 16:20:49.375330 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.375285    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5f4ac5-141c-4b00-869d-942cc8198b1d" containerName="revoke-keys"
Nov 03 16:20:49.375387 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.375378    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5f4ac5-141c-4b00-869d-942cc8198b1d" containerName="revoke-keys"
Nov 03 16:20:49.378871 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.378855    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"
Nov 03 16:20:49.385893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.385845    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:20:49.386020 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.385939    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:20:49.389025 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.389001    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"]
Nov 03 16:20:49.528188 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.528156    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae685f25-0167-4586-ba50-bf37f08ed5e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d7d6c3e-wxxjc\" (UID: \"ae685f25-0167-4586-ba50-bf37f08ed5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"
Nov 03 16:20:49.528363 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.528228    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae685f25-0167-4586-ba50-bf37f08ed5e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d7d6c3e-wxxjc\" (UID: \"ae685f25-0167-4586-ba50-bf37f08ed5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"
Nov 03 16:20:49.528408 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.528357    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfvkr\" (UniqueName: \"kubernetes.io/projected/ae685f25-0167-4586-ba50-bf37f08ed5e6-kube-api-access-tfvkr\") pod \"maas-api-revoke-sub-6d7d6c3e-wxxjc\" (UID: \"ae685f25-0167-4586-ba50-bf37f08ed5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"
Nov 03 16:20:49.629896 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.629821    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae685f25-0167-4586-ba50-bf37f08ed5e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d7d6c3e-wxxjc\" (UID: \"ae685f25-0167-4586-ba50-bf37f08ed5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"
Nov 03 16:20:49.629896 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.629881    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae685f25-0167-4586-ba50-bf37f08ed5e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d7d6c3e-wxxjc\" (UID: \"ae685f25-0167-4586-ba50-bf37f08ed5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"
Nov 03 16:20:49.630074 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.629943    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfvkr\" (UniqueName: \"kubernetes.io/projected/ae685f25-0167-4586-ba50-bf37f08ed5e6-kube-api-access-tfvkr\") pod \"maas-api-revoke-sub-6d7d6c3e-wxxjc\" (UID: \"ae685f25-0167-4586-ba50-bf37f08ed5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"
Nov 03 16:20:49.630534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.630508    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae685f25-0167-4586-ba50-bf37f08ed5e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d7d6c3e-wxxjc\" (UID: \"ae685f25-0167-4586-ba50-bf37f08ed5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"
Nov 03 16:20:49.632343 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.632325    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae685f25-0167-4586-ba50-bf37f08ed5e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d7d6c3e-wxxjc\" (UID: \"ae685f25-0167-4586-ba50-bf37f08ed5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"
Nov 03 16:20:49.642891 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.642870    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfvkr\" (UniqueName: \"kubernetes.io/projected/ae685f25-0167-4586-ba50-bf37f08ed5e6-kube-api-access-tfvkr\") pod \"maas-api-revoke-sub-6d7d6c3e-wxxjc\" (UID: \"ae685f25-0167-4586-ba50-bf37f08ed5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"
Nov 03 16:20:49.694364 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.694344    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"
Nov 03 16:20:49.917223 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:49.917090    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:50.016781 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:50.016752    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"]
Nov 03 16:20:50.017303 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:20:50.017279    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae685f25_0167_4586_ba50_bf37f08ed5e6.slice/crio-9aa59679cbe7f5ff4562ae093334280e280b456dfb2575bf5859cda0bf377380 WatchSource:0}: Error finding container 9aa59679cbe7f5ff4562ae093334280e280b456dfb2575bf5859cda0bf377380: Status 404 returned error can't find the container with id 9aa59679cbe7f5ff4562ae093334280e280b456dfb2575bf5859cda0bf377380
Nov 03 16:20:50.051746 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:50.051714    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc" event={"ID":"ae685f25-0167-4586-ba50-bf37f08ed5e6","Type":"ContainerStarted","Data":"9aa59679cbe7f5ff4562ae093334280e280b456dfb2575bf5859cda0bf377380"}
Nov 03 16:20:50.877309 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:50.877276    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:51.060687 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:51.060658    6506 generic.go:356] "Generic (PLEG): container finished" podID="ae685f25-0167-4586-ba50-bf37f08ed5e6" containerID="e6b388ec7a670c66516bcfd004850409af52e2bf582307e116b46c4b9926727c" exitCode=0
Nov 03 16:20:51.061053 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:51.060703    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc" event={"ID":"ae685f25-0167-4586-ba50-bf37f08ed5e6","Type":"ContainerDied","Data":"e6b388ec7a670c66516bcfd004850409af52e2bf582307e116b46c4b9926727c"}
Nov 03 16:20:51.685922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:51.685880    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:52.224865 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:52.224842    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"
Nov 03 16:20:52.268910 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:52.268881    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae685f25-0167-4586-ba50-bf37f08ed5e6-kube-api-access-tfvkr\" (UniqueName: \"kubernetes.io/projected/ae685f25-0167-4586-ba50-bf37f08ed5e6-kube-api-access-tfvkr\") pod \"ae685f25-0167-4586-ba50-bf37f08ed5e6\" (UID: \"ae685f25-0167-4586-ba50-bf37f08ed5e6\") "
Nov 03 16:20:52.269305 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:52.269255    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae685f25-0167-4586-ba50-bf37f08ed5e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae685f25-0167-4586-ba50-bf37f08ed5e6-maas-api-xks-ca\") pod \"ae685f25-0167-4586-ba50-bf37f08ed5e6\" (UID: \"ae685f25-0167-4586-ba50-bf37f08ed5e6\") "
Nov 03 16:20:52.269483 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:52.269386    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae685f25-0167-4586-ba50-bf37f08ed5e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae685f25-0167-4586-ba50-bf37f08ed5e6-maas-api-openshift-ca\") pod \"ae685f25-0167-4586-ba50-bf37f08ed5e6\" (UID: \"ae685f25-0167-4586-ba50-bf37f08ed5e6\") "
Nov 03 16:20:52.270127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:52.270100    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae685f25-0167-4586-ba50-bf37f08ed5e6-maas-api-openshift-ca" pod "ae685f25-0167-4586-ba50-bf37f08ed5e6" (UID: "ae685f25-0167-4586-ba50-bf37f08ed5e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:20:52.271862 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:52.271837    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae685f25-0167-4586-ba50-bf37f08ed5e6-maas-api-xks-ca" pod "ae685f25-0167-4586-ba50-bf37f08ed5e6" (UID: "ae685f25-0167-4586-ba50-bf37f08ed5e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:20:52.272530 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:52.272504    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae685f25-0167-4586-ba50-bf37f08ed5e6-kube-api-access-tfvkr" pod "ae685f25-0167-4586-ba50-bf37f08ed5e6" (UID: "ae685f25-0167-4586-ba50-bf37f08ed5e6"). InnerVolumeSpecName "kube-api-access-tfvkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:20:52.371175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:52.371096    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae685f25-0167-4586-ba50-bf37f08ed5e6-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:52.371175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:52.371131    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae685f25-0167-4586-ba50-bf37f08ed5e6-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:52.371175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:52.371146    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfvkr\" (UniqueName: \"kubernetes.io/projected/ae685f25-0167-4586-ba50-bf37f08ed5e6-kube-api-access-tfvkr\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:52.482132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:52.482095    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:53.075168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:53.075140    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"
Nov 03 16:20:53.075365 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:53.075139    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc" event={"ID":"ae685f25-0167-4586-ba50-bf37f08ed5e6","Type":"ContainerDied","Data":"9aa59679cbe7f5ff4562ae093334280e280b456dfb2575bf5859cda0bf377380"}
Nov 03 16:20:53.075365 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:53.075274    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aa59679cbe7f5ff4562ae093334280e280b456dfb2575bf5859cda0bf377380"
Nov 03 16:20:53.311278 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:53.311240    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:54.025323 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.025287    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"]
Nov 03 16:20:54.025870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.025852    6506 cpu_manager.go:414] "containerMap: removing container" podUID="ae685f25-0167-4586-ba50-bf37f08ed5e6" containerName="revoke-keys"
Nov 03 16:20:54.025870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.025867    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae685f25-0167-4586-ba50-bf37f08ed5e6" containerName="revoke-keys"
Nov 03 16:20:54.026052 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.025995    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae685f25-0167-4586-ba50-bf37f08ed5e6" containerName="revoke-keys"
Nov 03 16:20:54.029932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.029913    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.053167 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.053140    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"]
Nov 03 16:20:54.128912 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.126576    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:54.215523 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.215484    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46df749f-6606-477d-85db-cfe8e35bd740-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.215698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.215548    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.215698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.215670    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.215847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.215719    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.215929 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.215857    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.215929 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.215890    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46df749f-6606-477d-85db-cfe8e35bd740-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.216057 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.216003    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.216116 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.216101    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46df749f-6606-477d-85db-cfe8e35bd740-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.216166 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.216131    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsdh8\" (UniqueName: \"kubernetes.io/projected/46df749f-6606-477d-85db-cfe8e35bd740-kube-api-access-hsdh8\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.317349 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.317261    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46df749f-6606-477d-85db-cfe8e35bd740-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.317349 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.317317    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsdh8\" (UniqueName: \"kubernetes.io/projected/46df749f-6606-477d-85db-cfe8e35bd740-kube-api-access-hsdh8\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.317893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.317421    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46df749f-6606-477d-85db-cfe8e35bd740-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.317893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.317469    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.317893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.317533    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.317893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.317600    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.317893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.317648    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.317893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.317672    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46df749f-6606-477d-85db-cfe8e35bd740-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.317893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.317758    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.318272 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.318059    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.318272 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.318115    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.318272 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.318185    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.318450 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.318308    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.318667 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.318647    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46df749f-6606-477d-85db-cfe8e35bd740-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.320176 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.320154    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.320277 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.320239    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46df749f-6606-477d-85db-cfe8e35bd740-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.347638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.347610    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsdh8\" (UniqueName: \"kubernetes.io/projected/46df749f-6606-477d-85db-cfe8e35bd740-kube-api-access-hsdh8\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.371290 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.371269    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46df749f-6606-477d-85db-cfe8e35bd740-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.504534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.504491    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"]
Nov 03 16:20:54.509693 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.509669    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.532431 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.532406    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"]
Nov 03 16:20:54.611005 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.610936    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"]
Nov 03 16:20:54.615153 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.615127    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.623385 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.623345    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.623488 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.623410    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.623488 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.623459    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.623559 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.623534    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.623619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.623602    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.623748 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.623733    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0fa8f4-44df-45e7-8028-7d329877afaa-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.623864 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.623847    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.623934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.623920    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.624012 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.623995    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gnvn\" (UniqueName: \"kubernetes.io/projected/af0fa8f4-44df-45e7-8028-7d329877afaa-kube-api-access-8gnvn\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.638754 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.638724    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"]
Nov 03 16:20:54.642717 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.642702    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.645979 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.645943    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:54.653195 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.653170    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"]
Nov 03 16:20:54.684807 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.684704    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"]
Nov 03 16:20:54.727281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.726577    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0fa8f4-44df-45e7-8028-7d329877afaa-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.727281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.726610    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119e87d8-1b7b-46db-9ac8-289804f75184-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.727281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.726654    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.727281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.726668    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.727281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.726704    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.727281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.726722    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119e87d8-1b7b-46db-9ac8-289804f75184-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.727281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.726743    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.727281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.726765    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.727281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.726808    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gnvn\" (UniqueName: \"kubernetes.io/projected/af0fa8f4-44df-45e7-8028-7d329877afaa-kube-api-access-8gnvn\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.728018 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.727832    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119e87d8-1b7b-46db-9ac8-289804f75184-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.728018 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.727914    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.728018 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.727978    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.728168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.728034    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.728168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.728089    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.728274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.728164    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0fa8f4-44df-45e7-8028-7d329877afaa-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.728274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.728200    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.728386 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.728306    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.728386 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.728362    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.728499 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.728405    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qdkq\" (UniqueName: \"kubernetes.io/projected/119e87d8-1b7b-46db-9ac8-289804f75184-kube-api-access-5qdkq\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.728499 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.728457    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.728705 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.728686    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.728917 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.728899    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.729148 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.729130    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.729901 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.729859    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.748751 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.745059    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.748751 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.745533    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.757843 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.757784    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gnvn\" (UniqueName: \"kubernetes.io/projected/af0fa8f4-44df-45e7-8028-7d329877afaa-kube-api-access-8gnvn\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.824790 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.824755    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:54.829375 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.829356    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qdkq\" (UniqueName: \"kubernetes.io/projected/119e87d8-1b7b-46db-9ac8-289804f75184-kube-api-access-5qdkq\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.829430 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.829394    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b827t\" (UniqueName: \"kubernetes.io/projected/17b47389-9b79-402f-bc9b-d404e0b1e75c-kube-api-access-b827t\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.829483 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.829428    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119e87d8-1b7b-46db-9ac8-289804f75184-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.829483 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.829448    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.829583 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.829563    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.829642 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.829625    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.829704 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.829686    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.829760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.829720    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119e87d8-1b7b-46db-9ac8-289804f75184-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.829760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.829749    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.829894 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.829832    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.829894 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.829859    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b47389-9b79-402f-bc9b-d404e0b1e75c-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.830005 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.829903    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119e87d8-1b7b-46db-9ac8-289804f75184-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.830005 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.829938    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.830005 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.829993    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.830154 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.829998    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.830154 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.830036    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.830154 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.830045    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.830154 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.830071    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.830154 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.830108    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.830563 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.830262    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.830563 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.830322    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.830563 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.830485    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119e87d8-1b7b-46db-9ac8-289804f75184-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.830563 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.830506    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.832209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.832190    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119e87d8-1b7b-46db-9ac8-289804f75184-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.832293 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.832255    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.843437 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.843416    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qdkq\" (UniqueName: \"kubernetes.io/projected/119e87d8-1b7b-46db-9ac8-289804f75184-kube-api-access-5qdkq\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.846270 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.846246    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119e87d8-1b7b-46db-9ac8-289804f75184-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.930418 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.930386    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:54.931533 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.931500    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.931718 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.931697    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b47389-9b79-402f-bc9b-d404e0b1e75c-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.931930 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.931910    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.932057 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.932040    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.932191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.932174    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.932305 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.932290    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.932533 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.932514    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b827t\" (UniqueName: \"kubernetes.io/projected/17b47389-9b79-402f-bc9b-d404e0b1e75c-kube-api-access-b827t\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.932669 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.932339    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.932910 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.932680    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.933263 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.932579    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b47389-9b79-402f-bc9b-d404e0b1e75c-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.933263 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.932847    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.933263 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.932956    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.933263 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.933059    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.933470 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.933367    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.936052 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.936028    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.936336 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.936316    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.946292 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.946268    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b827t\" (UniqueName: \"kubernetes.io/projected/17b47389-9b79-402f-bc9b-d404e0b1e75c-kube-api-access-b827t\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.947362 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.947339    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.963092 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.963062    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"]
Nov 03 16:20:54.963190 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.963148    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:54.966193 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.966162    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:54.966273 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.966228    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:54.966273 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:54.966256    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:55.098258 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.098227    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q" event={"ID":"46df749f-6606-477d-85db-cfe8e35bd740","Type":"ContainerStarted","Data":"2f514dd7a56e21bd532836d14728f4f63d0fc089fe38336ceb6fa1abb996ff42"}
Nov 03 16:20:55.098400 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.098279    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q" event={"ID":"46df749f-6606-477d-85db-cfe8e35bd740","Type":"ContainerStarted","Data":"9f0619ffaf3d9f6e3be14f0ab7e5434d266e575d643b4f93065bd94b60c84571"}
Nov 03 16:20:55.130846 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.127567    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:55.130846 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.127645    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:55.130846 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.127695    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:55.130846 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.128247    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"]
Nov 03 16:20:55.132545 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.132204    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q" podStartSLOduration=2.1321867660000002 podStartE2EDuration="2.132186766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:55.125642085 +0000 UTC m=+1069.139040232" watchObservedRunningTime="2026-11-03 16:20:55.132186766 +0000 UTC m=+1069.145584913"
Nov 03 16:20:55.276471 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.276441    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"]
Nov 03 16:20:55.288565 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.288507    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:55.288843 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.288816    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:55.288998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.288980    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:55.310309 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.310220    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:55.375922 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:20:55.375888    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b47389_9b79_402f_bc9b_d404e0b1e75c.slice/crio-3dec0774310ed9c04d15862b7ee95beb8c7ef12dfbecbc35775951ad13b6a413 WatchSource:0}: Error finding container 3dec0774310ed9c04d15862b7ee95beb8c7ef12dfbecbc35775951ad13b6a413: Status 404 returned error can't find the container with id 3dec0774310ed9c04d15862b7ee95beb8c7ef12dfbecbc35775951ad13b6a413
Nov 03 16:20:55.378482 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.378452    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"]
Nov 03 16:20:55.380345 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.380304    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:55.380463 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.380418    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:55.380522 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.380459    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:20:55.446979 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.446908    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"]
Nov 03 16:20:55.451597 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.451577    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"
Nov 03 16:20:55.456967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.456949    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:20:55.457109 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.456983    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:20:55.465613 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.465590    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"]
Nov 03 16:20:55.550471 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.550048    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aefed321-8510-4ca9-a066-1cbe448bf763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13f65682-hdk9t\" (UID: \"aefed321-8510-4ca9-a066-1cbe448bf763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"
Nov 03 16:20:55.550471 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.550109    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aefed321-8510-4ca9-a066-1cbe448bf763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13f65682-hdk9t\" (UID: \"aefed321-8510-4ca9-a066-1cbe448bf763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"
Nov 03 16:20:55.550471 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.550388    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qj25\" (UniqueName: \"kubernetes.io/projected/aefed321-8510-4ca9-a066-1cbe448bf763-kube-api-access-9qj25\") pod \"maas-api-revoke-sub-13f65682-hdk9t\" (UID: \"aefed321-8510-4ca9-a066-1cbe448bf763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"
Nov 03 16:20:55.647931 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.646949    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:20:55.653287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.653256    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qj25\" (UniqueName: \"kubernetes.io/projected/aefed321-8510-4ca9-a066-1cbe448bf763-kube-api-access-9qj25\") pod \"maas-api-revoke-sub-13f65682-hdk9t\" (UID: \"aefed321-8510-4ca9-a066-1cbe448bf763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"
Nov 03 16:20:55.653562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.653544    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aefed321-8510-4ca9-a066-1cbe448bf763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13f65682-hdk9t\" (UID: \"aefed321-8510-4ca9-a066-1cbe448bf763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"
Nov 03 16:20:55.653698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.653683    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aefed321-8510-4ca9-a066-1cbe448bf763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13f65682-hdk9t\" (UID: \"aefed321-8510-4ca9-a066-1cbe448bf763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"
Nov 03 16:20:55.654703 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.654681    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aefed321-8510-4ca9-a066-1cbe448bf763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13f65682-hdk9t\" (UID: \"aefed321-8510-4ca9-a066-1cbe448bf763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"
Nov 03 16:20:55.662242 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.662219    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aefed321-8510-4ca9-a066-1cbe448bf763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13f65682-hdk9t\" (UID: \"aefed321-8510-4ca9-a066-1cbe448bf763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"
Nov 03 16:20:55.672365 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.672311    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qj25\" (UniqueName: \"kubernetes.io/projected/aefed321-8510-4ca9-a066-1cbe448bf763-kube-api-access-9qj25\") pod \"maas-api-revoke-sub-13f65682-hdk9t\" (UID: \"aefed321-8510-4ca9-a066-1cbe448bf763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"
Nov 03 16:20:55.789147 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:55.789116    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"
Nov 03 16:20:56.201876 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.198253    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"]
Nov 03 16:20:56.201876 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.198415    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml" event={"ID":"119e87d8-1b7b-46db-9ac8-289804f75184","Type":"ContainerStarted","Data":"1e606cb3a3438c6c9d3240e6ea43b8da8665f969a53943b64aa7ce083b54d288"}
Nov 03 16:20:56.201876 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.198549    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml" event={"ID":"119e87d8-1b7b-46db-9ac8-289804f75184","Type":"ContainerStarted","Data":"4ab06d501223496bae6d05c91fabd1c4a99c77bce816866d19f625ea6d75c2b3"}
Nov 03 16:20:56.205893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.204219    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556" event={"ID":"17b47389-9b79-402f-bc9b-d404e0b1e75c","Type":"ContainerStarted","Data":"b46da3dbf1aa72f92e8acec9d704c7c5b0d23b8513f3038f0e8e0eadfc51dc57"}
Nov 03 16:20:56.205893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.204298    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556" event={"ID":"17b47389-9b79-402f-bc9b-d404e0b1e75c","Type":"ContainerStarted","Data":"3dec0774310ed9c04d15862b7ee95beb8c7ef12dfbecbc35775951ad13b6a413"}
Nov 03 16:20:56.381302 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.306689    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml" podStartSLOduration=2.306671692 podStartE2EDuration="2.306671692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:56.306085395 +0000 UTC m=+1070.319483546" watchObservedRunningTime="2026-11-03 16:20:56.306671692 +0000 UTC m=+1070.320069840"
Nov 03 16:20:56.381302 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.371882    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j" event={"ID":"af0fa8f4-44df-45e7-8028-7d329877afaa","Type":"ContainerStarted","Data":"036dad6f01f50d4737e26f3850a56715895fa42e162ae6a7b82cd0d405d6809d"}
Nov 03 16:20:56.381302 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.371943    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j" event={"ID":"af0fa8f4-44df-45e7-8028-7d329877afaa","Type":"ContainerStarted","Data":"82f8e994eaad7c8821bb54be99e34d0899cf63b1a91b85ff0a86035aec42ce47"}
Nov 03 16:20:56.517907 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.512622    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:56.583727 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.573418    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556" podStartSLOduration=2.573396065 podStartE2EDuration="2.573396065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:56.538028542 +0000 UTC m=+1070.551426689" watchObservedRunningTime="2026-11-03 16:20:56.573396065 +0000 UTC m=+1070.586794212"
Nov 03 16:20:56.665830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.660234    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 16:20:56.665830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.660314    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q" podUID="46df749f-6606-477d-85db-cfe8e35bd740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 16:20:56.749460 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.749339    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:56.845171 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.845138    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:20:56.877845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.876930    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 16:20:56.877845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.876989    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j" podUID="af0fa8f4-44df-45e7-8028-7d329877afaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 16:20:56.998230 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.998193    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:20:56.998405 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.998271    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:20:57.009644 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.998606    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:57.009644 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.998604    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 16:20:57.009644 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.998653    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556" podUID="17b47389-9b79-402f-bc9b-d404e0b1e75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 16:20:57.009644 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:56.998680    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml" podUID="119e87d8-1b7b-46db-9ac8-289804f75184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 16:20:57.009644 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.006152    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j" podStartSLOduration=3.006133698 podStartE2EDuration="3.006133698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:56.634015542 +0000 UTC m=+1070.647413680" watchObservedRunningTime="2026-11-03 16:20:57.006133698 +0000 UTC m=+1071.019531845"
Nov 03 16:20:57.255421 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.254201    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:57.297466 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.297358    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:57.309642 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.309599    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:57.350002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.349931    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:57.416884 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.416848    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t" event={"ID":"aefed321-8510-4ca9-a066-1cbe448bf763","Type":"ContainerStarted","Data":"2721f154dd92bb57256caf0ba1a043f7343876daa40a1f36f4b94570d6c04fd8"}
Nov 03 16:20:57.514104 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.509208    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 16:20:57.530949 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.530573    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:57.607880 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.607836    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:57.663977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.663939    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 16:20:57.664890 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.664848    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q" podUID="46df749f-6606-477d-85db-cfe8e35bd740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 16:20:57.827923 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.825338    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 16:20:57.827923 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.825417    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j" podUID="af0fa8f4-44df-45e7-8028-7d329877afaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 16:20:57.942822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.940114    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 16:20:57.942822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.940192    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml" podUID="119e87d8-1b7b-46db-9ac8-289804f75184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 16:20:57.984822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.983879    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:57.984822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:57.983939    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556" podUID="17b47389-9b79-402f-bc9b-d404e0b1e75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 16:20:58.247863 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.244899    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 16:20:58.257007 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.256903    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:58.262829 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.257327    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:20:58.304823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.300870    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:58.304823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.301642    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:20:58.328822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.311866    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:58.328822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.312325    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:20:58.361009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.358869    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:20:58.361009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.359488    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:20:58.482874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.414916    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:58.620995 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.620954    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:58.622097 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.622071    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:20:58.623370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.623345    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 16:20:58.623833 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.623813    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:20:58.657949 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.657915    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 16:20:58.658206 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.658179    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q" podUID="46df749f-6606-477d-85db-cfe8e35bd740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 16:20:58.746099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.704277    6506 generic.go:356] "Generic (PLEG): container finished" podID="aefed321-8510-4ca9-a066-1cbe448bf763" containerID="c00bb1ceaa16a30289f62347599581dfb395e5d43f1a9b980a2e3bc22a0ddd90" exitCode=0
Nov 03 16:20:58.746099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.706530    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t" event={"ID":"aefed321-8510-4ca9-a066-1cbe448bf763","Type":"ContainerDied","Data":"c00bb1ceaa16a30289f62347599581dfb395e5d43f1a9b980a2e3bc22a0ddd90"}
Nov 03 16:20:58.884947 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.884911    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 16:20:58.885259 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:58.885230    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j" podUID="af0fa8f4-44df-45e7-8028-7d329877afaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 16:20:59.004518 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.004420    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 16:20:59.004763 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.004733    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml" podUID="119e87d8-1b7b-46db-9ac8-289804f75184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 16:20:59.031838 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.006224    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:59.032202 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.032160    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556" podUID="17b47389-9b79-402f-bc9b-d404e0b1e75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 16:20:59.202856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.171385    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:20:59.260882 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.256897    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:59.260882 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.258403    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:20:59.260882 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.258452    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:59.324821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.310911    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Nov 03 16:20:59.324821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.313062    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:59.382883 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.382451    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:59.421348 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:20:59.420993    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:20:59.502822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.495892    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:20:59.502822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.497135    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:20:59.642822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.633698    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 16:20:59.679099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.679063    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 16:20:59.698223 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.698188    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 16:20:59.698573 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.698530    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q" podUID="46df749f-6606-477d-85db-cfe8e35bd740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 16:20:59.924017 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.923918    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 16:20:59.924253 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.924226    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j" podUID="af0fa8f4-44df-45e7-8028-7d329877afaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 16:20:59.945946 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.941059    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 16:20:59.946134 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.945959    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml" podUID="119e87d8-1b7b-46db-9ac8-289804f75184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 16:20:59.966297 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.966265    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:59.991994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:20:59.991955    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556" podUID="17b47389-9b79-402f-bc9b-d404e0b1e75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 16:21:00.048762 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.048728    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:21:00.294877 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.272315    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:00.294877 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.273494    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:00.357841 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.328931    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:00.357841 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.329673    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Nov 03 16:21:00.614119 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.614007    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:00.695823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.692664    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:00.793522 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.778778    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:00.793522 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.785105    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:21:00.806825 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.804228    6506 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:00.818815 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.807737    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 16:21:00.838495 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.838454    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q" podUID="46df749f-6606-477d-85db-cfe8e35bd740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 16:21:00.844905 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.840903    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 16:21:00.845196 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.845165    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j" podUID="af0fa8f4-44df-45e7-8028-7d329877afaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 16:21:00.845358 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.836556    6506 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:00.869903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.864641    6506 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:00.966773 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.965982    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 16:21:00.966773 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.966041    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml" podUID="119e87d8-1b7b-46db-9ac8-289804f75184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 16:21:01.008846 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.991603    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:21:01.008846 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:00.991662    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556" podUID="17b47389-9b79-402f-bc9b-d404e0b1e75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 16:21:01.292823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:01.288854    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:01.320015 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:01.297743    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:01.356851 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:01.347901    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:01.356851 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:01.349129    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:01.478327 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:01.478265    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:21:01.656300 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:01.656201    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:21:01.753421 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:01.753365    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 16:21:01.756002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:01.755973    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q" podUID="46df749f-6606-477d-85db-cfe8e35bd740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 16:21:01.790979 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:01.755395    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 16:21:01.831911 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:01.806491    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:01.985876 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:01.985751    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 16:21:01.986137 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:01.986112    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:21:01.986285 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:01.986263    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556" podUID="17b47389-9b79-402f-bc9b-d404e0b1e75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 16:21:02.027301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.008911    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 16:21:02.027301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.008977    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j" podUID="af0fa8f4-44df-45e7-8028-7d329877afaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 16:21:02.027301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.016895    6506 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Nov 03 16:21:02.047946 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.047900    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml" podUID="119e87d8-1b7b-46db-9ac8-289804f75184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 16:21:02.311831 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.310104    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 16:21:02.311831 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.310677    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 16:21:02.359167 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.359111    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:02.364582 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.364351    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:02.364582 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.363915    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:02.383054 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.383013    6506 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:02.584973 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.545487    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:21:02.584973 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.551297    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:21:02.688822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.687739    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:02.689419 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.689390    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 16:21:02.689586 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.689564    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q" podUID="46df749f-6606-477d-85db-cfe8e35bd740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 16:21:02.784208 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.784171    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"
Nov 03 16:21:02.811264 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.793136    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:02.851873 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.814228    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:21:02.851873 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.816372    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:02.854163 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.854133    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 16:21:02.854375 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.854348    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j" podUID="af0fa8f4-44df-45e7-8028-7d329877afaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 16:21:02.892063 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.875972    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aefed321-8510-4ca9-a066-1cbe448bf763-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aefed321-8510-4ca9-a066-1cbe448bf763-maas-api-xks-ca\") pod \"aefed321-8510-4ca9-a066-1cbe448bf763\" (UID: \"aefed321-8510-4ca9-a066-1cbe448bf763\") "
Nov 03 16:21:02.892063 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.876195    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aefed321-8510-4ca9-a066-1cbe448bf763-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aefed321-8510-4ca9-a066-1cbe448bf763-maas-api-openshift-ca\") pod \"aefed321-8510-4ca9-a066-1cbe448bf763\" (UID: \"aefed321-8510-4ca9-a066-1cbe448bf763\") "
Nov 03 16:21:02.985750 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.985709    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aefed321-8510-4ca9-a066-1cbe448bf763-kube-api-access-9qj25\" (UniqueName: \"kubernetes.io/projected/aefed321-8510-4ca9-a066-1cbe448bf763-kube-api-access-9qj25\") pod \"aefed321-8510-4ca9-a066-1cbe448bf763\" (UID: \"aefed321-8510-4ca9-a066-1cbe448bf763\") "
Nov 03 16:21:02.986947 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:02.986915    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 16:21:03.010214 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.010167    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml" podUID="119e87d8-1b7b-46db-9ac8-289804f75184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 16:21:03.010845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.005055    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:21:03.011048 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.011026    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556" podUID="17b47389-9b79-402f-bc9b-d404e0b1e75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 16:21:03.181499 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.181413    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aefed321-8510-4ca9-a066-1cbe448bf763-maas-api-openshift-ca" pod "aefed321-8510-4ca9-a066-1cbe448bf763" (UID: "aefed321-8510-4ca9-a066-1cbe448bf763"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:03.199636 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.199505    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aefed321-8510-4ca9-a066-1cbe448bf763-maas-api-xks-ca" pod "aefed321-8510-4ca9-a066-1cbe448bf763" (UID: "aefed321-8510-4ca9-a066-1cbe448bf763"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:03.243260 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.242927    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:21:03.269868 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.245372    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefed321-8510-4ca9-a066-1cbe448bf763-kube-api-access-9qj25" pod "aefed321-8510-4ca9-a066-1cbe448bf763" (UID: "aefed321-8510-4ca9-a066-1cbe448bf763"). InnerVolumeSpecName "kube-api-access-9qj25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:03.270127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.260179    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aefed321-8510-4ca9-a066-1cbe448bf763-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:03.270252 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.270237    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aefed321-8510-4ca9-a066-1cbe448bf763-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:03.272189 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.265329    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:03.325145 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.325103    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 16:21:03.329625 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.325360    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:03.364201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.364036    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:03.365156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.364595    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:03.365727 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.364604    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:21:03.416874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.407582    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qj25\" (UniqueName: \"kubernetes.io/projected/aefed321-8510-4ca9-a066-1cbe448bf763-kube-api-access-9qj25\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:03.449344 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.434571    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:21:03.481947 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.478305    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:21:03.483295 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.483045    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"
Nov 03 16:21:03.484148 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.483173    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t" event={"ID":"aefed321-8510-4ca9-a066-1cbe448bf763","Type":"ContainerDied","Data":"2721f154dd92bb57256caf0ba1a043f7343876daa40a1f36f4b94570d6c04fd8"}
Nov 03 16:21:03.484148 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.483488    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2721f154dd92bb57256caf0ba1a043f7343876daa40a1f36f4b94570d6c04fd8"
Nov 03 16:21:03.579592 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.579563    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:21:03.603059 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.603026    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:21:03.652029 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.649761    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:21:03.666071 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.666044    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:21:03.832136 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.832101    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 16:21:03.832669 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.832638    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j" podUID="af0fa8f4-44df-45e7-8028-7d329877afaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 16:21:03.931346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.931263    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 16:21:03.931893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.931708    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml" podUID="119e87d8-1b7b-46db-9ac8-289804f75184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 16:21:03.963562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.963523    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:21:03.963717 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:03.963579    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556" podUID="17b47389-9b79-402f-bc9b-d404e0b1e75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 16:21:04.393455 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:04.393372    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:04.458586 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:04.458557    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:21:04.459218 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:04.459085    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:21:04.461240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:04.460722    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:21:05.249818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:05.249024    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:21:05.250318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:05.250045    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:21:05.259816 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:05.254528    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:21:05.343622 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:05.340562    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:05.467399 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:05.467362    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:21:05.467781 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:05.467757    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:21:05.469505 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:05.469486    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:21:05.566580 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:05.566552    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:21:05.568403 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:05.568382    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:21:05.570012 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:05.569993    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:21:06.104075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:06.104034    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:07.016588 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:07.016541    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:07.831178 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:07.831145    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:08.825509 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:08.825477    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:09.366455 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:09.366423    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:09.649933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:09.649831    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:10.326911 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:10.314896    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:10.375823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:10.355244    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 16:21:10.741880 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:10.731914    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:10.811324 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:10.811260    6506 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:11.305822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:11.304884    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:11.325820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:11.324463    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:21:11.325820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:11.324866    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 16:21:11.325820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:11.325512    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:21:11.408822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:11.384898    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:11.408822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:11.385694    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:21:11.408822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:11.407398    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:11.449819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:11.442266    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:11.736876 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:11.735991    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:11.736876 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:11.736382    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:21:11.795821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:11.792153    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:21:11.919931 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:11.919887    6506 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:12.317820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:12.314909    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 16:21:12.333683 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:12.330761    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:21:12.336822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:12.334275    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:12.336822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:12.334508    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:21:12.336822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:12.334717    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:21:12.461249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:12.460449    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:12.461249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:12.460789    6506 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded"
Nov 03 16:21:12.461249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:12.460927    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:21:12.488855 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:12.478595    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 16:21:12.599999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:12.495926    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:12.620822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:12.601334    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:21:12.818821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:12.793837    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 16:21:12.819568 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:12.819530    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:21:12.820007 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:12.819989    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:21:13.317636 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:13.317585    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:13.430907 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:13.409983    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:13.459851 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:13.446841    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 16:21:13.473941 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:13.473894    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:13.606996 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:13.606890    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:13.607675 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:13.607227    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:13.824993 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:13.824938    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:13.826080 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:13.824971    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:14.375818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:14.369950    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:14.481405 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:14.458232    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:14.481405 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:14.460091    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:14.483387 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:14.483362    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:21:14.523904 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:14.508687    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:14.601395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:14.601354    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:14.613054 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:14.613007    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:14.613610 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:14.613582    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Nov 03 16:21:14.820818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:14.806877    6506 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:14.842821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:14.828974    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"]
Nov 03 16:21:14.842821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:14.831368    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:14.842821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:14.831692    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:21:14.885816 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:14.873702    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:21:15.077870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.076344    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:21:15.105478 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.104232    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:21:15.117231 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.117206    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:21:15.133193 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.133168    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:21:15.135680 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.135661    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:21:15.137641 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.137622    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:21:15.882093 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.882014    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"]
Nov 03 16:21:15.882878 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.882856    6506 cpu_manager.go:414] "containerMap: removing container" podUID="aefed321-8510-4ca9-a066-1cbe448bf763" containerName="revoke-keys"
Nov 03 16:21:15.882985 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.882878    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aefed321-8510-4ca9-a066-1cbe448bf763" containerName="revoke-keys"
Nov 03 16:21:15.883051 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.883021    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="aefed321-8510-4ca9-a066-1cbe448bf763" containerName="revoke-keys"
Nov 03 16:21:15.888019 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.887997    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"
Nov 03 16:21:15.892355 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.892324    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:21:15.892472 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.892426    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:21:15.896362 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.896241    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"]
Nov 03 16:21:15.921809 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.921761    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac85523a-362a-4087-80c1-1db0c26d5f13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ea9526b-nqp7z\" (UID: \"ac85523a-362a-4087-80c1-1db0c26d5f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"
Nov 03 16:21:15.922011 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.921989    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9726\" (UniqueName: \"kubernetes.io/projected/ac85523a-362a-4087-80c1-1db0c26d5f13-kube-api-access-l9726\") pod \"maas-api-revoke-sub-2ea9526b-nqp7z\" (UID: \"ac85523a-362a-4087-80c1-1db0c26d5f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"
Nov 03 16:21:15.922148 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:15.922128    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac85523a-362a-4087-80c1-1db0c26d5f13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ea9526b-nqp7z\" (UID: \"ac85523a-362a-4087-80c1-1db0c26d5f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"
Nov 03 16:21:16.022990 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:16.022953    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac85523a-362a-4087-80c1-1db0c26d5f13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ea9526b-nqp7z\" (UID: \"ac85523a-362a-4087-80c1-1db0c26d5f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"
Nov 03 16:21:16.023165 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:16.023073    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac85523a-362a-4087-80c1-1db0c26d5f13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ea9526b-nqp7z\" (UID: \"ac85523a-362a-4087-80c1-1db0c26d5f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"
Nov 03 16:21:16.023165 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:16.023153    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9726\" (UniqueName: \"kubernetes.io/projected/ac85523a-362a-4087-80c1-1db0c26d5f13-kube-api-access-l9726\") pod \"maas-api-revoke-sub-2ea9526b-nqp7z\" (UID: \"ac85523a-362a-4087-80c1-1db0c26d5f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"
Nov 03 16:21:16.023716 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:16.023691    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac85523a-362a-4087-80c1-1db0c26d5f13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ea9526b-nqp7z\" (UID: \"ac85523a-362a-4087-80c1-1db0c26d5f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"
Nov 03 16:21:16.025555 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:16.025534    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac85523a-362a-4087-80c1-1db0c26d5f13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ea9526b-nqp7z\" (UID: \"ac85523a-362a-4087-80c1-1db0c26d5f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"
Nov 03 16:21:16.035697 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:16.035677    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9726\" (UniqueName: \"kubernetes.io/projected/ac85523a-362a-4087-80c1-1db0c26d5f13-kube-api-access-l9726\") pod \"maas-api-revoke-sub-2ea9526b-nqp7z\" (UID: \"ac85523a-362a-4087-80c1-1db0c26d5f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"
Nov 03 16:21:16.202976 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:16.202910    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"
Nov 03 16:21:16.504155 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:16.504124    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"]
Nov 03 16:21:16.505183 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:21:16.505155    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac85523a_362a_4087_80c1_1db0c26d5f13.slice/crio-c254b18fdf97a01c3133c416ead6b1bc01045c3d958a2f044f842ae31d392ec9 WatchSource:0}: Error finding container c254b18fdf97a01c3133c416ead6b1bc01045c3d958a2f044f842ae31d392ec9: Status 404 returned error can't find the container with id c254b18fdf97a01c3133c416ead6b1bc01045c3d958a2f044f842ae31d392ec9
Nov 03 16:21:16.695124 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:16.695072    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:17.184927 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:17.184828    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z" event={"ID":"ac85523a-362a-4087-80c1-1db0c26d5f13","Type":"ContainerStarted","Data":"c265582b80f2804620a220b3b026ff703abcd793cc14ca8ad578c0f5b36b83fd"}
Nov 03 16:21:17.184927 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:17.184893    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z" event={"ID":"ac85523a-362a-4087-80c1-1db0c26d5f13","Type":"ContainerStarted","Data":"c254b18fdf97a01c3133c416ead6b1bc01045c3d958a2f044f842ae31d392ec9"}
Nov 03 16:21:17.215722 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:17.215674    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z" podStartSLOduration=2.215662149 podStartE2EDuration="2.215662149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:17.211451624 +0000 UTC m=+1091.224849751" watchObservedRunningTime="2026-11-03 16:21:17.215662149 +0000 UTC m=+1091.229060295"
Nov 03 16:21:17.897639 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:17.897601    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:18.193945 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:18.193858    6506 generic.go:356] "Generic (PLEG): container finished" podID="ac85523a-362a-4087-80c1-1db0c26d5f13" containerID="c265582b80f2804620a220b3b026ff703abcd793cc14ca8ad578c0f5b36b83fd" exitCode=0
Nov 03 16:21:18.194307 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:18.193955    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z" event={"ID":"ac85523a-362a-4087-80c1-1db0c26d5f13","Type":"ContainerDied","Data":"c265582b80f2804620a220b3b026ff703abcd793cc14ca8ad578c0f5b36b83fd"}
Nov 03 16:21:18.836378 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:18.836346    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:19.397283 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:19.397261    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"
Nov 03 16:21:19.471514 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:19.471433    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac85523a-362a-4087-80c1-1db0c26d5f13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac85523a-362a-4087-80c1-1db0c26d5f13-maas-api-openshift-ca\") pod \"ac85523a-362a-4087-80c1-1db0c26d5f13\" (UID: \"ac85523a-362a-4087-80c1-1db0c26d5f13\") "
Nov 03 16:21:19.471673 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:19.471566    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac85523a-362a-4087-80c1-1db0c26d5f13-kube-api-access-l9726\" (UniqueName: \"kubernetes.io/projected/ac85523a-362a-4087-80c1-1db0c26d5f13-kube-api-access-l9726\") pod \"ac85523a-362a-4087-80c1-1db0c26d5f13\" (UID: \"ac85523a-362a-4087-80c1-1db0c26d5f13\") "
Nov 03 16:21:19.471673 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:19.471656    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac85523a-362a-4087-80c1-1db0c26d5f13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac85523a-362a-4087-80c1-1db0c26d5f13-maas-api-xks-ca\") pod \"ac85523a-362a-4087-80c1-1db0c26d5f13\" (UID: \"ac85523a-362a-4087-80c1-1db0c26d5f13\") "
Nov 03 16:21:19.471836 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:19.471789    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac85523a-362a-4087-80c1-1db0c26d5f13-maas-api-openshift-ca" pod "ac85523a-362a-4087-80c1-1db0c26d5f13" (UID: "ac85523a-362a-4087-80c1-1db0c26d5f13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:19.472384 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:19.472364    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac85523a-362a-4087-80c1-1db0c26d5f13-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:19.474205 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:19.474139    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac85523a-362a-4087-80c1-1db0c26d5f13-kube-api-access-l9726" pod "ac85523a-362a-4087-80c1-1db0c26d5f13" (UID: "ac85523a-362a-4087-80c1-1db0c26d5f13"). InnerVolumeSpecName "kube-api-access-l9726". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:19.474320 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:19.474255    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac85523a-362a-4087-80c1-1db0c26d5f13-maas-api-xks-ca" pod "ac85523a-362a-4087-80c1-1db0c26d5f13" (UID: "ac85523a-362a-4087-80c1-1db0c26d5f13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:19.573349 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:19.573319    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9726\" (UniqueName: \"kubernetes.io/projected/ac85523a-362a-4087-80c1-1db0c26d5f13-kube-api-access-l9726\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:19.573349 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:19.573347    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac85523a-362a-4087-80c1-1db0c26d5f13-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:19.847868 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:19.847789    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:20.212937 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:20.212909    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"
Nov 03 16:21:20.213094 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:20.212902    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z" event={"ID":"ac85523a-362a-4087-80c1-1db0c26d5f13","Type":"ContainerDied","Data":"c254b18fdf97a01c3133c416ead6b1bc01045c3d958a2f044f842ae31d392ec9"}
Nov 03 16:21:20.213094 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:20.213032    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c254b18fdf97a01c3133c416ead6b1bc01045c3d958a2f044f842ae31d392ec9"
Nov 03 16:21:20.403733 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:20.403655    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"]
Nov 03 16:21:20.515200 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:20.515166    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:20.680880 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:20.680476    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:20.854652 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:20.854588    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:21.591003 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.590975    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:21.693708 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.693675    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"]
Nov 03 16:21:21.694299 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.694285    6506 cpu_manager.go:414] "containerMap: removing container" podUID="ac85523a-362a-4087-80c1-1db0c26d5f13" containerName="revoke-keys"
Nov 03 16:21:21.694351 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.694300    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac85523a-362a-4087-80c1-1db0c26d5f13" containerName="revoke-keys"
Nov 03 16:21:21.694404 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.694396    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac85523a-362a-4087-80c1-1db0c26d5f13" containerName="revoke-keys"
Nov 03 16:21:21.697683 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.697668    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"
Nov 03 16:21:21.704699 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.704674    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:21:21.704889 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.704768    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:21:21.710726 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.710698    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"]
Nov 03 16:21:21.762836 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.762746    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:21.812743 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.812710    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115bede2-738c-48f7-af23-efd9bf52c022-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts\" (UID: \"115bede2-738c-48f7-af23-efd9bf52c022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"
Nov 03 16:21:21.813003 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.812762    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnhm\" (UniqueName: \"kubernetes.io/projected/115bede2-738c-48f7-af23-efd9bf52c022-kube-api-access-flnhm\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts\" (UID: \"115bede2-738c-48f7-af23-efd9bf52c022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"
Nov 03 16:21:21.813070 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.812989    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115bede2-738c-48f7-af23-efd9bf52c022-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts\" (UID: \"115bede2-738c-48f7-af23-efd9bf52c022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"
Nov 03 16:21:21.914139 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.914060    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnhm\" (UniqueName: \"kubernetes.io/projected/115bede2-738c-48f7-af23-efd9bf52c022-kube-api-access-flnhm\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts\" (UID: \"115bede2-738c-48f7-af23-efd9bf52c022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"
Nov 03 16:21:21.914293 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.914172    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115bede2-738c-48f7-af23-efd9bf52c022-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts\" (UID: \"115bede2-738c-48f7-af23-efd9bf52c022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"
Nov 03 16:21:21.914372 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.914355    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115bede2-738c-48f7-af23-efd9bf52c022-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts\" (UID: \"115bede2-738c-48f7-af23-efd9bf52c022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"
Nov 03 16:21:21.915071 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.915005    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115bede2-738c-48f7-af23-efd9bf52c022-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts\" (UID: \"115bede2-738c-48f7-af23-efd9bf52c022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"
Nov 03 16:21:21.916737 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.916716    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115bede2-738c-48f7-af23-efd9bf52c022-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts\" (UID: \"115bede2-738c-48f7-af23-efd9bf52c022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"
Nov 03 16:21:21.927477 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:21.927451    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnhm\" (UniqueName: \"kubernetes.io/projected/115bede2-738c-48f7-af23-efd9bf52c022-kube-api-access-flnhm\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts\" (UID: \"115bede2-738c-48f7-af23-efd9bf52c022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"
Nov 03 16:21:22.034224 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:22.034187    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"
Nov 03 16:21:22.338712 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:22.338681    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"]
Nov 03 16:21:22.339130 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:21:22.339105    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115bede2_738c_48f7_af23_efd9bf52c022.slice/crio-e1e33266eb4e2cc4242a7b3f0343440d169583f5c731250453229f46b26d44ce WatchSource:0}: Error finding container e1e33266eb4e2cc4242a7b3f0343440d169583f5c731250453229f46b26d44ce: Status 404 returned error can't find the container with id e1e33266eb4e2cc4242a7b3f0343440d169583f5c731250453229f46b26d44ce
Nov 03 16:21:22.793899 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:22.791295    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:23.237814 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:23.237765    6506 generic.go:356] "Generic (PLEG): container finished" podID="115bede2-738c-48f7-af23-efd9bf52c022" containerID="b7aa383af6563f09b3ae8d4c306f21d0a5e80b06d43b673fb72d672f4b98a4d0" exitCode=0
Nov 03 16:21:23.237965 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:23.237832    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts" event={"ID":"115bede2-738c-48f7-af23-efd9bf52c022","Type":"ContainerDied","Data":"b7aa383af6563f09b3ae8d4c306f21d0a5e80b06d43b673fb72d672f4b98a4d0"}
Nov 03 16:21:23.237965 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:23.237886    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts" event={"ID":"115bede2-738c-48f7-af23-efd9bf52c022","Type":"ContainerStarted","Data":"e1e33266eb4e2cc4242a7b3f0343440d169583f5c731250453229f46b26d44ce"}
Nov 03 16:21:23.686687 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:23.686649    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:23.962546 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:23.962467    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"]
Nov 03 16:21:23.967001 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:23.966894    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:23.981871 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:23.981848    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"]
Nov 03 16:21:24.043463 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.043432    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.043619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.043492    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.043619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.043563    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd6d999-549b-4cbd-920a-39660ee9c30f-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.043619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.043588    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.043776 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.043693    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.043776 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.043713    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.043935 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.043913    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc87k\" (UniqueName: \"kubernetes.io/projected/abd6d999-549b-4cbd-920a-39660ee9c30f-kube-api-access-nc87k\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.043988 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.043970    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.044056 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.044042    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.145721 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.145685    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc87k\" (UniqueName: \"kubernetes.io/projected/abd6d999-549b-4cbd-920a-39660ee9c30f-kube-api-access-nc87k\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.145933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.145728    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.145933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.145771    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.145933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.145845    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.145933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.145885    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.145933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.145922    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd6d999-549b-4cbd-920a-39660ee9c30f-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.146203 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.145953    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.146203 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.146039    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.146203 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.146062    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.146361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.146197    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.146361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.146243    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.146361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.146351    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.146685 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.146605    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd6d999-549b-4cbd-920a-39660ee9c30f-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.146946 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.146789    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.148344 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.148323    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.148620 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.148604    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.158895 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.158871    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc87k\" (UniqueName: \"kubernetes.io/projected/abd6d999-549b-4cbd-920a-39660ee9c30f-kube-api-access-nc87k\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.159864 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.159843    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.283662 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.283621    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:24.330512 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.329517    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"]
Nov 03 16:21:24.343100 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.343075    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.350861 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.350830    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"]
Nov 03 16:21:24.429456 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.429429    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"
Nov 03 16:21:24.456842 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.456789    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.457006 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.456980    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.457156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.457080    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.457156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.457147    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.457290 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.457184    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfhjd\" (UniqueName: \"kubernetes.io/projected/c81a5fc9-15c0-491f-bb7a-939e4485f544-kube-api-access-vfhjd\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.457352 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.457307    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.457413 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.457379    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.457463 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.457417    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81a5fc9-15c0-491f-bb7a-939e4485f544-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.457519 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.457491    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.528822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.524010    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:24.558491 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.558455    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/115bede2-738c-48f7-af23-efd9bf52c022-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115bede2-738c-48f7-af23-efd9bf52c022-maas-api-xks-ca\") pod \"115bede2-738c-48f7-af23-efd9bf52c022\" (UID: \"115bede2-738c-48f7-af23-efd9bf52c022\") "
Nov 03 16:21:24.558662 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.558638    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115bede2-738c-48f7-af23-efd9bf52c022-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115bede2-738c-48f7-af23-efd9bf52c022-maas-api-openshift-ca\") pod \"115bede2-738c-48f7-af23-efd9bf52c022\" (UID: \"115bede2-738c-48f7-af23-efd9bf52c022\") "
Nov 03 16:21:24.558721 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.558706    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115bede2-738c-48f7-af23-efd9bf52c022-kube-api-access-flnhm\" (UniqueName: \"kubernetes.io/projected/115bede2-738c-48f7-af23-efd9bf52c022-kube-api-access-flnhm\") pod \"115bede2-738c-48f7-af23-efd9bf52c022\" (UID: \"115bede2-738c-48f7-af23-efd9bf52c022\") "
Nov 03 16:21:24.558987 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.558963    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115bede2-738c-48f7-af23-efd9bf52c022-maas-api-openshift-ca" pod "115bede2-738c-48f7-af23-efd9bf52c022" (UID: "115bede2-738c-48f7-af23-efd9bf52c022"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:24.559094 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.559005    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.559094 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.559037    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfhjd\" (UniqueName: \"kubernetes.io/projected/c81a5fc9-15c0-491f-bb7a-939e4485f544-kube-api-access-vfhjd\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.559203 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.559122    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.559203 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.559177    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.559203 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.559194    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81a5fc9-15c0-491f-bb7a-939e4485f544-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.559370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.559283    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.559370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.559361    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.559613 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.559484    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.559613 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.559490    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.559613 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.559478    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.559613 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.559574    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.559836 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.559666    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115bede2-738c-48f7-af23-efd9bf52c022-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:24.559985 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.559961    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.560216 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.560197    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.560704 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.560652    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81a5fc9-15c0-491f-bb7a-939e4485f544-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.561283 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.561247    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/115bede2-738c-48f7-af23-efd9bf52c022-maas-api-xks-ca" pod "115bede2-738c-48f7-af23-efd9bf52c022" (UID: "115bede2-738c-48f7-af23-efd9bf52c022"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:24.561721 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.561699    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115bede2-738c-48f7-af23-efd9bf52c022-kube-api-access-flnhm" pod "115bede2-738c-48f7-af23-efd9bf52c022" (UID: "115bede2-738c-48f7-af23-efd9bf52c022"). InnerVolumeSpecName "kube-api-access-flnhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:24.561947 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.561923    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.562126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.562108    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.572144 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.572121    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.572574 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.572556    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfhjd\" (UniqueName: \"kubernetes.io/projected/c81a5fc9-15c0-491f-bb7a-939e4485f544-kube-api-access-vfhjd\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.613069 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.612862    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"]
Nov 03 16:21:24.620643 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:21:24.620599    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd6d999_549b_4cbd_920a_39660ee9c30f.slice/crio-c82ca920fbae28952fdf4da0a826bd0f8e6a98eda5412056e62dbc6b0783a244 WatchSource:0}: Error finding container c82ca920fbae28952fdf4da0a826bd0f8e6a98eda5412056e62dbc6b0783a244: Status 404 returned error can't find the container with id c82ca920fbae28952fdf4da0a826bd0f8e6a98eda5412056e62dbc6b0783a244
Nov 03 16:21:24.624081 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.624036    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:21:24.624218 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.624127    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:21:24.624218 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.624178    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:21:24.661402 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.661375    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115bede2-738c-48f7-af23-efd9bf52c022-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:24.661402 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.661403    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flnhm\" (UniqueName: \"kubernetes.io/projected/115bede2-738c-48f7-af23-efd9bf52c022-kube-api-access-flnhm\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:24.670718 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.670691    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:24.921424 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.921177    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"]
Nov 03 16:21:24.921940 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.921918    6506 cpu_manager.go:414] "containerMap: removing container" podUID="115bede2-738c-48f7-af23-efd9bf52c022" containerName="revoke-keys"
Nov 03 16:21:24.921940 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.921939    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115bede2-738c-48f7-af23-efd9bf52c022" containerName="revoke-keys"
Nov 03 16:21:24.922114 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.922105    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="115bede2-738c-48f7-af23-efd9bf52c022" containerName="revoke-keys"
Nov 03 16:21:24.926262 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.926240    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:24.941516 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.940315    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"]
Nov 03 16:21:24.985646 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.985604    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:21:24.986538 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.986502    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:21:24.986643 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.986566    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:21:24.990191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:24.990167    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"]
Nov 03 16:21:25.068206 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.068180    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.068306 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.068243    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwtmr\" (UniqueName: \"kubernetes.io/projected/c01949d1-06c5-4f23-855d-6e3b04d0abe8-kube-api-access-hwtmr\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.068410 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.068387    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.068549 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.068530    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.068619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.068594    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.068743 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.068644    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.068743 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.068700    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.068921 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.068819    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.068921 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.068845    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.183415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.179654    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.183415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.179705    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.183415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.179845    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.183415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.179883    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.183415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.179978    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.183415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.180075    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwtmr\" (UniqueName: \"kubernetes.io/projected/c01949d1-06c5-4f23-855d-6e3b04d0abe8-kube-api-access-hwtmr\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.183415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.180165    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.183415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.180290    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.183415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.180333    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.183415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.180810    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.183415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.181070    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.183415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.181328    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.185149 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.185116    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.185267 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.185151    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.188382 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.188323    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.193157 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.193090    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.203372 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.203343    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.205504 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.205480    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwtmr\" (UniqueName: \"kubernetes.io/projected/c01949d1-06c5-4f23-855d-6e3b04d0abe8-kube-api-access-hwtmr\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.252623 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.252576    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:25.258326 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.257832    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg" event={"ID":"c81a5fc9-15c0-491f-bb7a-939e4485f544","Type":"ContainerStarted","Data":"03d73ada7084ffd770703c631df0f207ae30072fab9a450fa4e19f6a156c425d"}
Nov 03 16:21:25.258326 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.257891    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg" event={"ID":"c81a5fc9-15c0-491f-bb7a-939e4485f544","Type":"ContainerStarted","Data":"b91a9eddc0113c57541dc9ae70154d27279b1cb2532a389d4e8e3dd2fc59f7c8"}
Nov 03 16:21:25.259405 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.259387    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"
Nov 03 16:21:25.259405 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.259395    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts" event={"ID":"115bede2-738c-48f7-af23-efd9bf52c022","Type":"ContainerDied","Data":"e1e33266eb4e2cc4242a7b3f0343440d169583f5c731250453229f46b26d44ce"}
Nov 03 16:21:25.259551 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.259437    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1e33266eb4e2cc4242a7b3f0343440d169583f5c731250453229f46b26d44ce"
Nov 03 16:21:25.261472 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.261452    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr" event={"ID":"abd6d999-549b-4cbd-920a-39660ee9c30f","Type":"ContainerStarted","Data":"b7cc9d22c2b1c15109a5f732679e9ca916b81119e51ae158a8a7247f373c09a9"}
Nov 03 16:21:25.261602 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.261487    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr" event={"ID":"abd6d999-549b-4cbd-920a-39660ee9c30f","Type":"ContainerStarted","Data":"c82ca920fbae28952fdf4da0a826bd0f8e6a98eda5412056e62dbc6b0783a244"}
Nov 03 16:21:25.288172 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.288121    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:25.289453 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.289431    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 16:21:25.289564 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.289475    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr" podUID="abd6d999-549b-4cbd-920a-39660ee9c30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 16:21:25.292051 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.292005    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg" podStartSLOduration=1.291992341 podStartE2EDuration="1.291992341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:25.287277446 +0000 UTC m=+1099.300675591" watchObservedRunningTime="2026-11-03 16:21:25.291992341 +0000 UTC m=+1099.305390487"
Nov 03 16:21:25.344817 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.344528    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr" podStartSLOduration=2.344507037 podStartE2EDuration="2.344507037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:25.329392981 +0000 UTC m=+1099.342791130" watchObservedRunningTime="2026-11-03 16:21:25.344507037 +0000 UTC m=+1099.357905184"
Nov 03 16:21:25.404515 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.404476    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt" podUID="a78c2dad-d700-4697-9220-c08a07a3d4c3" containerName="payload-pre-processing" containerID="cri-o://aeeff5f3f449f200493af9d617a18fe75e0fbb61fb59a2fb8eba8ee68de0a604" gracePeriod=25
Nov 03 16:21:25.528876 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.522474    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:25.611094 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.611061    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"]
Nov 03 16:21:25.623745 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.622785    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:21:25.623745 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.622922    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:21:25.623745 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.622968    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:21:25.671032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:25.671008    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:26.286925 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.283943    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 16:21:26.287415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.286960    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr" podUID="abd6d999-549b-4cbd-920a-39660ee9c30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 16:21:26.290193 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.290165    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj" event={"ID":"c01949d1-06c5-4f23-855d-6e3b04d0abe8","Type":"ContainerStarted","Data":"c85d52c611d3516814cd30257a4c8fbc6d697d216808fe4ae6b851b34d652a35"}
Nov 03 16:21:26.290388 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.290376    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj" event={"ID":"c01949d1-06c5-4f23-855d-6e3b04d0abe8","Type":"ContainerStarted","Data":"f49cae83479927cf443707d35dcaaf0d296a9621fbaddb1bd3913ba87591df70"}
Nov 03 16:21:26.369660 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.369488    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj" podStartSLOduration=2.369465718 podStartE2EDuration="2.369465718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:26.328133766 +0000 UTC m=+1100.341531929" watchObservedRunningTime="2026-11-03 16:21:26.369465718 +0000 UTC m=+1100.382863867"
Nov 03 16:21:26.507821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.506438    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"]
Nov 03 16:21:26.615878 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.597266    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"]
Nov 03 16:21:26.615878 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.597390    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"
Nov 03 16:21:26.661830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.634178    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:21:26.661830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.634309    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:21:26.754821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.740297    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930af4d2-a222-47b3-b6d8-3aff6206f248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-787a6067-wnw2r\" (UID: \"930af4d2-a222-47b3-b6d8-3aff6206f248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"
Nov 03 16:21:26.754821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.740369    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bqgs\" (UniqueName: \"kubernetes.io/projected/930af4d2-a222-47b3-b6d8-3aff6206f248-kube-api-access-8bqgs\") pod \"maas-api-revoke-sub-787a6067-wnw2r\" (UID: \"930af4d2-a222-47b3-b6d8-3aff6206f248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"
Nov 03 16:21:26.754821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.740401    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930af4d2-a222-47b3-b6d8-3aff6206f248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-787a6067-wnw2r\" (UID: \"930af4d2-a222-47b3-b6d8-3aff6206f248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"
Nov 03 16:21:26.754821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.752720    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:21:26.754821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.752785    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg" podUID="c81a5fc9-15c0-491f-bb7a-939e4485f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 16:21:26.788827 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.779215    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:26.861842 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.843286    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930af4d2-a222-47b3-b6d8-3aff6206f248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-787a6067-wnw2r\" (UID: \"930af4d2-a222-47b3-b6d8-3aff6206f248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"
Nov 03 16:21:26.861842 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.843350    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bqgs\" (UniqueName: \"kubernetes.io/projected/930af4d2-a222-47b3-b6d8-3aff6206f248-kube-api-access-8bqgs\") pod \"maas-api-revoke-sub-787a6067-wnw2r\" (UID: \"930af4d2-a222-47b3-b6d8-3aff6206f248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"
Nov 03 16:21:26.861842 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.843379    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930af4d2-a222-47b3-b6d8-3aff6206f248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-787a6067-wnw2r\" (UID: \"930af4d2-a222-47b3-b6d8-3aff6206f248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"
Nov 03 16:21:26.880879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.866548    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930af4d2-a222-47b3-b6d8-3aff6206f248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-787a6067-wnw2r\" (UID: \"930af4d2-a222-47b3-b6d8-3aff6206f248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"
Nov 03 16:21:26.893833 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.893325    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930af4d2-a222-47b3-b6d8-3aff6206f248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-787a6067-wnw2r\" (UID: \"930af4d2-a222-47b3-b6d8-3aff6206f248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"
Nov 03 16:21:26.902099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:26.901628    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bqgs\" (UniqueName: \"kubernetes.io/projected/930af4d2-a222-47b3-b6d8-3aff6206f248-kube-api-access-8bqgs\") pod \"maas-api-revoke-sub-787a6067-wnw2r\" (UID: \"930af4d2-a222-47b3-b6d8-3aff6206f248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"
Nov 03 16:21:27.100977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.100941    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"
Nov 03 16:21:27.258906 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.257387    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:27.258906 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.257976    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:27.271835 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.268581    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 16:21:27.271835 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.268647    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj" podUID="c01949d1-06c5-4f23-855d-6e3b04d0abe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 16:21:27.289179 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.289144    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 16:21:27.289788 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.289744    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr" podUID="abd6d999-549b-4cbd-920a-39660ee9c30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 16:21:27.504184 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.499828    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"]
Nov 03 16:21:27.504184 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.500671    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:27.512196 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.512172    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"
Nov 03 16:21:27.543821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.524489    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5553aa2c-597b-4bbb-8559-b97e50c48bd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m\" (UID: \"5553aa2c-597b-4bbb-8559-b97e50c48bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"
Nov 03 16:21:27.543821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.524851    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfm6z\" (UniqueName: \"kubernetes.io/projected/5553aa2c-597b-4bbb-8559-b97e50c48bd4-kube-api-access-sfm6z\") pod \"maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m\" (UID: \"5553aa2c-597b-4bbb-8559-b97e50c48bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"
Nov 03 16:21:27.543821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.525018    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5553aa2c-597b-4bbb-8559-b97e50c48bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m\" (UID: \"5553aa2c-597b-4bbb-8559-b97e50c48bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"
Nov 03 16:21:27.586900 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.578894    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"]
Nov 03 16:21:27.681820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.668187    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5553aa2c-597b-4bbb-8559-b97e50c48bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m\" (UID: \"5553aa2c-597b-4bbb-8559-b97e50c48bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"
Nov 03 16:21:27.681820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.668305    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5553aa2c-597b-4bbb-8559-b97e50c48bd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m\" (UID: \"5553aa2c-597b-4bbb-8559-b97e50c48bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"
Nov 03 16:21:27.681820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.668478    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfm6z\" (UniqueName: \"kubernetes.io/projected/5553aa2c-597b-4bbb-8559-b97e50c48bd4-kube-api-access-sfm6z\") pod \"maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m\" (UID: \"5553aa2c-597b-4bbb-8559-b97e50c48bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"
Nov 03 16:21:27.681820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.670008    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5553aa2c-597b-4bbb-8559-b97e50c48bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m\" (UID: \"5553aa2c-597b-4bbb-8559-b97e50c48bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"
Nov 03 16:21:27.700610 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.694834    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:21:27.700610 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.694883    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg" podUID="c81a5fc9-15c0-491f-bb7a-939e4485f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 16:21:27.785145 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.785103    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfm6z\" (UniqueName: \"kubernetes.io/projected/5553aa2c-597b-4bbb-8559-b97e50c48bd4-kube-api-access-sfm6z\") pod \"maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m\" (UID: \"5553aa2c-597b-4bbb-8559-b97e50c48bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"
Nov 03 16:21:27.800230 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.800189    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5553aa2c-597b-4bbb-8559-b97e50c48bd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m\" (UID: \"5553aa2c-597b-4bbb-8559-b97e50c48bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"
Nov 03 16:21:27.891676 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.889269    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"
Nov 03 16:21:27.999046 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:27.999022    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"]
Nov 03 16:21:28.001647 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:21:28.001613    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930af4d2_a222_47b3_b6d8_3aff6206f248.slice/crio-e8735efd37213514e1bce7e0ee97e84ab802dae7ba50a6965050210b7cb2353e WatchSource:0}: Error finding container e8735efd37213514e1bce7e0ee97e84ab802dae7ba50a6965050210b7cb2353e: Status 404 returned error can't find the container with id e8735efd37213514e1bce7e0ee97e84ab802dae7ba50a6965050210b7cb2353e
Nov 03 16:21:28.273820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:28.271992    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 16:21:28.273820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:28.272070    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj" podUID="c01949d1-06c5-4f23-855d-6e3b04d0abe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 16:21:28.297651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:28.297603    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 16:21:28.299156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:28.298486    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr" podUID="abd6d999-549b-4cbd-920a-39660ee9c30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 16:21:28.504855 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:28.504312    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r" event={"ID":"930af4d2-a222-47b3-b6d8-3aff6206f248","Type":"ContainerStarted","Data":"e8735efd37213514e1bce7e0ee97e84ab802dae7ba50a6965050210b7cb2353e"}
Nov 03 16:21:28.672287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:28.672251    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:21:28.679911 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:28.679876    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg" podUID="c81a5fc9-15c0-491f-bb7a-939e4485f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 16:21:28.688233 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:28.688200    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"]
Nov 03 16:21:29.000758 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:21:29.000695    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:21:29.264782 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:29.264275    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 16:21:29.264782 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:29.264338    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj" podUID="c01949d1-06c5-4f23-855d-6e3b04d0abe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 16:21:29.289211 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:29.288649    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 16:21:29.289211 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:29.288703    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr" podUID="abd6d999-549b-4cbd-920a-39660ee9c30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 16:21:29.461347 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:29.461312    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:29.695657 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:29.695605    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:21:29.696721 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:29.696688    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg" podUID="c81a5fc9-15c0-491f-bb7a-939e4485f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 16:21:29.940190 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:29.940092    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m" event={"ID":"5553aa2c-597b-4bbb-8559-b97e50c48bd4","Type":"ContainerStarted","Data":"d972433f8be6db243ebd133cd8443363647a8b4ea77de8340f8c329d745d2c43"}
Nov 03 16:21:29.940445 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:29.940422    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m" event={"ID":"5553aa2c-597b-4bbb-8559-b97e50c48bd4","Type":"ContainerStarted","Data":"cb17cd5742f18cfd67f8813994232965e345634e2bb65800fc62ec5d8623c62d"}
Nov 03 16:21:29.945859 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:29.945517    6506 generic.go:356] "Generic (PLEG): container finished" podID="930af4d2-a222-47b3-b6d8-3aff6206f248" containerID="8fd1a3569b5d68968e07a737e9a56d8c114bcb8e8f7e039f8e9e25e6f7825c30" exitCode=0
Nov 03 16:21:29.945859 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:29.945695    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r" event={"ID":"930af4d2-a222-47b3-b6d8-3aff6206f248","Type":"ContainerDied","Data":"8fd1a3569b5d68968e07a737e9a56d8c114bcb8e8f7e039f8e9e25e6f7825c30"}
Nov 03 16:21:29.969536 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:29.969443    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main" containerID="cri-o://01663f319542374ffb8c089a265a598191d28da35f82cc345fdc424186562336" gracePeriod=45
Nov 03 16:21:30.002096 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:30.000541    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m" podStartSLOduration=3.000522454 podStartE2EDuration="3.000522454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:29.972233537 +0000 UTC m=+1103.985631685" watchObservedRunningTime="2026-11-03 16:21:30.000522454 +0000 UTC m=+1104.013920601"
Nov 03 16:21:30.089391 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:30.083395    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:30.142981 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:30.142939    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": dial tcp 10.133.0.56:8000: connect: connection refused"
Nov 03 16:21:30.303178 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:30.303143    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 16:21:30.303415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:30.303389    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr" podUID="abd6d999-549b-4cbd-920a-39660ee9c30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 16:21:30.378709 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:30.378454    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 16:21:30.378709 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:30.378514    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj" podUID="c01949d1-06c5-4f23-855d-6e3b04d0abe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 16:21:30.451923 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:30.451879    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:30.500198 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:30.458974    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:30.503844 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:30.500940    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:30.703832 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:30.701888    6506 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:30.758224 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:30.756085    6506 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:30.758224 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:30.757678    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:21:30.758224 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:30.757722    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg" podUID="c81a5fc9-15c0-491f-bb7a-939e4485f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 16:21:31.081180 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.081146    6506 generic.go:356] "Generic (PLEG): container finished" podID="5553aa2c-597b-4bbb-8559-b97e50c48bd4" containerID="d972433f8be6db243ebd133cd8443363647a8b4ea77de8340f8c329d745d2c43" exitCode=0
Nov 03 16:21:31.081433 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.081412    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m" event={"ID":"5553aa2c-597b-4bbb-8559-b97e50c48bd4","Type":"ContainerDied","Data":"d972433f8be6db243ebd133cd8443363647a8b4ea77de8340f8c329d745d2c43"}
Nov 03 16:21:31.084742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.084710    6506 generic.go:356] "Generic (PLEG): container finished" podID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerID="01663f319542374ffb8c089a265a598191d28da35f82cc345fdc424186562336" exitCode=0
Nov 03 16:21:31.085438 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.085046    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" event={"ID":"ad1e4d9f-21b8-40a7-a919-c53ce248510e","Type":"ContainerDied","Data":"01663f319542374ffb8c089a265a598191d28da35f82cc345fdc424186562336"}
Nov 03 16:21:31.259873 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.259065    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 16:21:31.259873 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.259122    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj" podUID="c01949d1-06c5-4f23-855d-6e3b04d0abe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 16:21:31.285406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.284109    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 16:21:31.285406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.284163    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr" podUID="abd6d999-549b-4cbd-920a-39660ee9c30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 16:21:31.333423 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.333044    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:21:31.468008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.421162    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad1e4d9f-21b8-40a7-a919-c53ce248510e-kube-api-access-kckvk\" (UniqueName: \"kubernetes.io/projected/ad1e4d9f-21b8-40a7-a919-c53ce248510e-kube-api-access-kckvk\") pod \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") "
Nov 03 16:21:31.468008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.421352    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-dshm\") pod \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") "
Nov 03 16:21:31.468008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.421388    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad1e4d9f-21b8-40a7-a919-c53ce248510e-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad1e4d9f-21b8-40a7-a919-c53ce248510e-tls-certs\") pod \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") "
Nov 03 16:21:31.468008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.421461    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-model-cache\") pod \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") "
Nov 03 16:21:31.468008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.421498    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-tmp-dir\") pod \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") "
Nov 03 16:21:31.468008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.421724    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-home\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-home\") pod \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\" (UID: \"ad1e4d9f-21b8-40a7-a919-c53ce248510e\") "
Nov 03 16:21:31.468008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.437564    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-model-cache" pod "ad1e4d9f-21b8-40a7-a919-c53ce248510e" (UID: "ad1e4d9f-21b8-40a7-a919-c53ce248510e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:31.468008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.437756    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-tmp-dir" pod "ad1e4d9f-21b8-40a7-a919-c53ce248510e" (UID: "ad1e4d9f-21b8-40a7-a919-c53ce248510e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:31.468008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.467945    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-home" pod "ad1e4d9f-21b8-40a7-a919-c53ce248510e" (UID: "ad1e4d9f-21b8-40a7-a919-c53ce248510e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:31.478813 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.476477    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1e4d9f-21b8-40a7-a919-c53ce248510e-kube-api-access-kckvk" pod "ad1e4d9f-21b8-40a7-a919-c53ce248510e" (UID: "ad1e4d9f-21b8-40a7-a919-c53ce248510e"). InnerVolumeSpecName "kube-api-access-kckvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:31.481697 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.481639    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad1e4d9f-21b8-40a7-a919-c53ce248510e-tls-certs" pod "ad1e4d9f-21b8-40a7-a919-c53ce248510e" (UID: "ad1e4d9f-21b8-40a7-a919-c53ce248510e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:31.486085 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.486031    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-dshm" pod "ad1e4d9f-21b8-40a7-a919-c53ce248510e" (UID: "ad1e4d9f-21b8-40a7-a919-c53ce248510e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:31.559353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.559208    6506 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-tmp-dir\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:31.559353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.559248    6506 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-home\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:31.559353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.559265    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kckvk\" (UniqueName: \"kubernetes.io/projected/ad1e4d9f-21b8-40a7-a919-c53ce248510e-kube-api-access-kckvk\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:31.559353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.559280    6506 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-dshm\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:31.559353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.559297    6506 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad1e4d9f-21b8-40a7-a919-c53ce248510e-tls-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:31.559353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.559311    6506 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad1e4d9f-21b8-40a7-a919-c53ce248510e-model-cache\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:31.648381 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.648036    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"
Nov 03 16:21:31.672819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.672371    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:21:31.672819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.672429    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg" podUID="c81a5fc9-15c0-491f-bb7a-939e4485f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 16:21:31.708820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.707789    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:31.764662 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.764628    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/930af4d2-a222-47b3-b6d8-3aff6206f248-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930af4d2-a222-47b3-b6d8-3aff6206f248-maas-api-xks-ca\") pod \"930af4d2-a222-47b3-b6d8-3aff6206f248\" (UID: \"930af4d2-a222-47b3-b6d8-3aff6206f248\") "
Nov 03 16:21:31.764883 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.764681    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/930af4d2-a222-47b3-b6d8-3aff6206f248-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930af4d2-a222-47b3-b6d8-3aff6206f248-maas-api-openshift-ca\") pod \"930af4d2-a222-47b3-b6d8-3aff6206f248\" (UID: \"930af4d2-a222-47b3-b6d8-3aff6206f248\") "
Nov 03 16:21:31.764883 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.764881    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/930af4d2-a222-47b3-b6d8-3aff6206f248-kube-api-access-8bqgs\" (UniqueName: \"kubernetes.io/projected/930af4d2-a222-47b3-b6d8-3aff6206f248-kube-api-access-8bqgs\") pod \"930af4d2-a222-47b3-b6d8-3aff6206f248\" (UID: \"930af4d2-a222-47b3-b6d8-3aff6206f248\") "
Nov 03 16:21:31.766963 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.766926    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930af4d2-a222-47b3-b6d8-3aff6206f248-maas-api-openshift-ca" pod "930af4d2-a222-47b3-b6d8-3aff6206f248" (UID: "930af4d2-a222-47b3-b6d8-3aff6206f248"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:31.769546 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.769519    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930af4d2-a222-47b3-b6d8-3aff6206f248-kube-api-access-8bqgs" pod "930af4d2-a222-47b3-b6d8-3aff6206f248" (UID: "930af4d2-a222-47b3-b6d8-3aff6206f248"). InnerVolumeSpecName "kube-api-access-8bqgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:31.769656 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.769565    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930af4d2-a222-47b3-b6d8-3aff6206f248-maas-api-xks-ca" pod "930af4d2-a222-47b3-b6d8-3aff6206f248" (UID: "930af4d2-a222-47b3-b6d8-3aff6206f248"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:31.867858 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.867774    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bqgs\" (UniqueName: \"kubernetes.io/projected/930af4d2-a222-47b3-b6d8-3aff6206f248-kube-api-access-8bqgs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:31.867858 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.867854    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930af4d2-a222-47b3-b6d8-3aff6206f248-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:31.868178 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:31.867873    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930af4d2-a222-47b3-b6d8-3aff6206f248-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:32.098595 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.098511    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:21:32.102676 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.102442    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"
Nov 03 16:21:32.147640 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.147221    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 16:21:32.151995 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.151957    6506 status_manager.go:453] "Container readiness changed before pod has synced" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" containerID="cri-o://01663f319542374ffb8c089a265a598191d28da35f82cc345fdc424186562336"
Nov 03 16:21:32.152216 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.152199    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"
Nov 03 16:21:32.152352 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.152332    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5" event={"ID":"ad1e4d9f-21b8-40a7-a919-c53ce248510e","Type":"ContainerDied","Data":"33dd168bea80448824018aacbc3274eb1a2feb0500189d3568163dfdd4b50908"}
Nov 03 16:21:32.152484 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.152469    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r" event={"ID":"930af4d2-a222-47b3-b6d8-3aff6206f248","Type":"ContainerDied","Data":"e8735efd37213514e1bce7e0ee97e84ab802dae7ba50a6965050210b7cb2353e"}
Nov 03 16:21:32.152620 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.152606    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8735efd37213514e1bce7e0ee97e84ab802dae7ba50a6965050210b7cb2353e"
Nov 03 16:21:32.152734 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.152722    6506 scope.go:122] "RemoveContainer" containerID="01663f319542374ffb8c089a265a598191d28da35f82cc345fdc424186562336"
Nov 03 16:21:32.170548 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.170145    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"]
Nov 03 16:21:32.179000 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.178975    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/simulator-w0-d569fb-kserve-587868d9d9-mflh5"]
Nov 03 16:21:32.252039 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.246251    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"]
Nov 03 16:21:32.258880 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.257910    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 16:21:32.258880 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.257972    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj" podUID="c01949d1-06c5-4f23-855d-6e3b04d0abe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 16:21:32.300714 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.284736    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 16:21:32.301067 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.301027    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr" podUID="abd6d999-549b-4cbd-920a-39660ee9c30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 16:21:32.451010 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.450970    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:32.482635 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.482461    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"
Nov 03 16:21:32.607621 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.607581    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5553aa2c-597b-4bbb-8559-b97e50c48bd4-kube-api-access-sfm6z\" (UniqueName: \"kubernetes.io/projected/5553aa2c-597b-4bbb-8559-b97e50c48bd4-kube-api-access-sfm6z\") pod \"5553aa2c-597b-4bbb-8559-b97e50c48bd4\" (UID: \"5553aa2c-597b-4bbb-8559-b97e50c48bd4\") "
Nov 03 16:21:32.608298 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.608266    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5553aa2c-597b-4bbb-8559-b97e50c48bd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5553aa2c-597b-4bbb-8559-b97e50c48bd4-maas-api-xks-ca\") pod \"5553aa2c-597b-4bbb-8559-b97e50c48bd4\" (UID: \"5553aa2c-597b-4bbb-8559-b97e50c48bd4\") "
Nov 03 16:21:32.609240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.608434    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5553aa2c-597b-4bbb-8559-b97e50c48bd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5553aa2c-597b-4bbb-8559-b97e50c48bd4-maas-api-openshift-ca\") pod \"5553aa2c-597b-4bbb-8559-b97e50c48bd4\" (UID: \"5553aa2c-597b-4bbb-8559-b97e50c48bd4\") "
Nov 03 16:21:32.609240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.608920    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5553aa2c-597b-4bbb-8559-b97e50c48bd4-maas-api-openshift-ca" pod "5553aa2c-597b-4bbb-8559-b97e50c48bd4" (UID: "5553aa2c-597b-4bbb-8559-b97e50c48bd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:32.609240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.609164    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5553aa2c-597b-4bbb-8559-b97e50c48bd4-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:32.613964 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.613907    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5553aa2c-597b-4bbb-8559-b97e50c48bd4-maas-api-xks-ca" pod "5553aa2c-597b-4bbb-8559-b97e50c48bd4" (UID: "5553aa2c-597b-4bbb-8559-b97e50c48bd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:32.614112 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.613967    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5553aa2c-597b-4bbb-8559-b97e50c48bd4-kube-api-access-sfm6z" pod "5553aa2c-597b-4bbb-8559-b97e50c48bd4" (UID: "5553aa2c-597b-4bbb-8559-b97e50c48bd4"). InnerVolumeSpecName "kube-api-access-sfm6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:32.657659 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.657515    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:32.674304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.674271    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:21:32.674560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.674530    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg" podUID="c81a5fc9-15c0-491f-bb7a-939e4485f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 16:21:32.712102 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.712063    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5553aa2c-597b-4bbb-8559-b97e50c48bd4-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:32.712102 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.712103    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfm6z\" (UniqueName: \"kubernetes.io/projected/5553aa2c-597b-4bbb-8559-b97e50c48bd4-kube-api-access-sfm6z\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:32.975760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:32.975640    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt" podUID="a78c2dad-d700-4697-9220-c08a07a3d4c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:21:33.168125 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:33.168085    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"
Nov 03 16:21:33.168777 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:33.168165    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m" event={"ID":"5553aa2c-597b-4bbb-8559-b97e50c48bd4","Type":"ContainerDied","Data":"cb17cd5742f18cfd67f8813994232965e345634e2bb65800fc62ec5d8623c62d"}
Nov 03 16:21:33.168935 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:33.168853    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb17cd5742f18cfd67f8813994232965e345634e2bb65800fc62ec5d8623c62d"
Nov 03 16:21:33.254634 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:33.254549    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 16:21:33.254634 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:33.254604    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj" podUID="c01949d1-06c5-4f23-855d-6e3b04d0abe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 16:21:33.296258 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:33.296229    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:33.296653 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:33.296629    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:33.297411 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:33.297389    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:21:33.858670 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:33.858634    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:34.106878 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.106849    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" path="/var/lib/kubelet/pods/ad1e4d9f-21b8-40a7-a919-c53ce248510e/volumes"
Nov 03 16:21:34.260714 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.260634    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:34.261226 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.261207    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:34.262096 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.262076    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:34.697740 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.697698    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:34.713033 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.712990    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:21:34.713228 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.713139    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg" podUID="c81a5fc9-15c0-491f-bb7a-939e4485f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:34.874070 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.874024    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:34.953997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.953914    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:34.954805 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.954774    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:34.956126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.956099    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:21:34.976005 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.975976    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"]
Nov 03 16:21:34.976722 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.976701    6506 cpu_manager.go:414] "containerMap: removing container" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main"
Nov 03 16:21:34.976722 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.976721    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main"
Nov 03 16:21:34.976884 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.976748    6506 cpu_manager.go:414] "containerMap: removing container" podUID="5553aa2c-597b-4bbb-8559-b97e50c48bd4" containerName="revoke-keys"
Nov 03 16:21:34.976884 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.976753    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5553aa2c-597b-4bbb-8559-b97e50c48bd4" containerName="revoke-keys"
Nov 03 16:21:34.976884 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.976778    6506 cpu_manager.go:414] "containerMap: removing container" podUID="930af4d2-a222-47b3-b6d8-3aff6206f248" containerName="revoke-keys"
Nov 03 16:21:34.976884 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.976782    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="930af4d2-a222-47b3-b6d8-3aff6206f248" containerName="revoke-keys"
Nov 03 16:21:34.977008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.976891    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="5553aa2c-597b-4bbb-8559-b97e50c48bd4" containerName="revoke-keys"
Nov 03 16:21:34.977008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.976905    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="930af4d2-a222-47b3-b6d8-3aff6206f248" containerName="revoke-keys"
Nov 03 16:21:34.977008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.976915    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad1e4d9f-21b8-40a7-a919-c53ce248510e" containerName="main"
Nov 03 16:21:34.980620 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.980598    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"
Nov 03 16:21:34.989609 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.989422    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:21:34.990983 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.990958    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"]
Nov 03 16:21:34.992974 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.992952    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:21:34.996958 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:34.996934    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"]
Nov 03 16:21:35.045905 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.045870    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"]
Nov 03 16:21:35.095517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.095483    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4deb4ddf-30da-4eb3-b56a-237274761339-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj\" (UID: \"4deb4ddf-30da-4eb3-b56a-237274761339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"
Nov 03 16:21:35.095906 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.095880    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4deb4ddf-30da-4eb3-b56a-237274761339-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj\" (UID: \"4deb4ddf-30da-4eb3-b56a-237274761339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"
Nov 03 16:21:35.096055 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.096026    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwtj8\" (UniqueName: \"kubernetes.io/projected/4deb4ddf-30da-4eb3-b56a-237274761339-kube-api-access-gwtj8\") pod \"maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj\" (UID: \"4deb4ddf-30da-4eb3-b56a-237274761339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"
Nov 03 16:21:35.133201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.129486    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:35.197173 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.197141    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4deb4ddf-30da-4eb3-b56a-237274761339-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj\" (UID: \"4deb4ddf-30da-4eb3-b56a-237274761339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"
Nov 03 16:21:35.197329 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.197183    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4deb4ddf-30da-4eb3-b56a-237274761339-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj\" (UID: \"4deb4ddf-30da-4eb3-b56a-237274761339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"
Nov 03 16:21:35.197329 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.197205    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwtj8\" (UniqueName: \"kubernetes.io/projected/4deb4ddf-30da-4eb3-b56a-237274761339-kube-api-access-gwtj8\") pod \"maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj\" (UID: \"4deb4ddf-30da-4eb3-b56a-237274761339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"
Nov 03 16:21:35.197779 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.197756    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4deb4ddf-30da-4eb3-b56a-237274761339-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj\" (UID: \"4deb4ddf-30da-4eb3-b56a-237274761339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"
Nov 03 16:21:35.199686 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.199663    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4deb4ddf-30da-4eb3-b56a-237274761339-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj\" (UID: \"4deb4ddf-30da-4eb3-b56a-237274761339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"
Nov 03 16:21:35.209758 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.209703    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwtj8\" (UniqueName: \"kubernetes.io/projected/4deb4ddf-30da-4eb3-b56a-237274761339-kube-api-access-gwtj8\") pod \"maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj\" (UID: \"4deb4ddf-30da-4eb3-b56a-237274761339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"
Nov 03 16:21:35.314471 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.314442    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"
Nov 03 16:21:35.646497 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.646467    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"]
Nov 03 16:21:35.646774 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:21:35.646748    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4deb4ddf_30da_4eb3_b56a_237274761339.slice/crio-c3d8e580f4608310387d7078d17bffd3040f25fff8f9d61a11ebc550d4a3c251 WatchSource:0}: Error finding container c3d8e580f4608310387d7078d17bffd3040f25fff8f9d61a11ebc550d4a3c251: Status 404 returned error can't find the container with id c3d8e580f4608310387d7078d17bffd3040f25fff8f9d61a11ebc550d4a3c251
Nov 03 16:21:35.698377 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.698349    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"]
Nov 03 16:21:35.704472 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.704449    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"
Nov 03 16:21:35.790221 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.790193    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"]
Nov 03 16:21:35.806566 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.806535    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgl26\" (UniqueName: \"kubernetes.io/projected/854b1626-e3b6-43dc-82a2-d83512df2718-kube-api-access-lgl26\") pod \"maas-api-revoke-sub-4cf8d3d4-mdl7j\" (UID: \"854b1626-e3b6-43dc-82a2-d83512df2718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"
Nov 03 16:21:35.806738 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.806719    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/854b1626-e3b6-43dc-82a2-d83512df2718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cf8d3d4-mdl7j\" (UID: \"854b1626-e3b6-43dc-82a2-d83512df2718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"
Nov 03 16:21:35.806968 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.806946    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/854b1626-e3b6-43dc-82a2-d83512df2718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cf8d3d4-mdl7j\" (UID: \"854b1626-e3b6-43dc-82a2-d83512df2718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"
Nov 03 16:21:35.908317 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.908245    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgl26\" (UniqueName: \"kubernetes.io/projected/854b1626-e3b6-43dc-82a2-d83512df2718-kube-api-access-lgl26\") pod \"maas-api-revoke-sub-4cf8d3d4-mdl7j\" (UID: \"854b1626-e3b6-43dc-82a2-d83512df2718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"
Nov 03 16:21:35.908728 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.908371    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/854b1626-e3b6-43dc-82a2-d83512df2718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cf8d3d4-mdl7j\" (UID: \"854b1626-e3b6-43dc-82a2-d83512df2718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"
Nov 03 16:21:35.908728 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.908545    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/854b1626-e3b6-43dc-82a2-d83512df2718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cf8d3d4-mdl7j\" (UID: \"854b1626-e3b6-43dc-82a2-d83512df2718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"
Nov 03 16:21:35.909131 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.909101    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/854b1626-e3b6-43dc-82a2-d83512df2718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cf8d3d4-mdl7j\" (UID: \"854b1626-e3b6-43dc-82a2-d83512df2718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"
Nov 03 16:21:35.911172 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:35.911153    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/854b1626-e3b6-43dc-82a2-d83512df2718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cf8d3d4-mdl7j\" (UID: \"854b1626-e3b6-43dc-82a2-d83512df2718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"
Nov 03 16:21:36.007610 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:36.007581    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgl26\" (UniqueName: \"kubernetes.io/projected/854b1626-e3b6-43dc-82a2-d83512df2718-kube-api-access-lgl26\") pod \"maas-api-revoke-sub-4cf8d3d4-mdl7j\" (UID: \"854b1626-e3b6-43dc-82a2-d83512df2718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"
Nov 03 16:21:36.020283 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:36.020259    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"
Nov 03 16:21:36.096180 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:36.095847    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:36.199155 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:36.199068    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj" event={"ID":"4deb4ddf-30da-4eb3-b56a-237274761339","Type":"ContainerStarted","Data":"9e1a174b6fa72b59cddbab456112bca984ca67b94b65670c1c6d7a6fb153f8e1"}
Nov 03 16:21:36.199155 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:36.199132    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj" event={"ID":"4deb4ddf-30da-4eb3-b56a-237274761339","Type":"ContainerStarted","Data":"c3d8e580f4608310387d7078d17bffd3040f25fff8f9d61a11ebc550d4a3c251"}
Nov 03 16:21:36.220272 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:36.220221    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj" podStartSLOduration=2.220207651 podStartE2EDuration="2.220207651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:36.218638004 +0000 UTC m=+1110.232036151" watchObservedRunningTime="2026-11-03 16:21:36.220207651 +0000 UTC m=+1110.233605797"
Nov 03 16:21:36.350773 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:21:36.350741    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854b1626_e3b6_43dc_82a2_d83512df2718.slice/crio-1cc753a03b381a5abdfaa5c47a4cc2383604140d3f6f113490c4bffc4fc2593e WatchSource:0}: Error finding container 1cc753a03b381a5abdfaa5c47a4cc2383604140d3f6f113490c4bffc4fc2593e: Status 404 returned error can't find the container with id 1cc753a03b381a5abdfaa5c47a4cc2383604140d3f6f113490c4bffc4fc2593e
Nov 03 16:21:36.360559 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:36.360534    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"]
Nov 03 16:21:36.836290 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:36.836212    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:37.055534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:37.055497    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:37.227954 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:37.227886    6506 generic.go:356] "Generic (PLEG): container finished" podID="4deb4ddf-30da-4eb3-b56a-237274761339" containerID="9e1a174b6fa72b59cddbab456112bca984ca67b94b65670c1c6d7a6fb153f8e1" exitCode=0
Nov 03 16:21:37.228197 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:37.228142    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj" event={"ID":"4deb4ddf-30da-4eb3-b56a-237274761339","Type":"ContainerDied","Data":"9e1a174b6fa72b59cddbab456112bca984ca67b94b65670c1c6d7a6fb153f8e1"}
Nov 03 16:21:37.230766 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:37.230735    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j" event={"ID":"854b1626-e3b6-43dc-82a2-d83512df2718","Type":"ContainerStarted","Data":"e6c2a3e3780191ca8989710c1eeae264827b5ececc471118eb27bb73a66b252c"}
Nov 03 16:21:37.230893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:37.230789    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j" event={"ID":"854b1626-e3b6-43dc-82a2-d83512df2718","Type":"ContainerStarted","Data":"1cc753a03b381a5abdfaa5c47a4cc2383604140d3f6f113490c4bffc4fc2593e"}
Nov 03 16:21:37.290397 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:37.290336    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j" podStartSLOduration=2.290315636 podStartE2EDuration="2.290315636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:37.284501714 +0000 UTC m=+1111.297899861" watchObservedRunningTime="2026-11-03 16:21:37.290315636 +0000 UTC m=+1111.303713782"
Nov 03 16:21:37.384210 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:37.384166    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:37.554394 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:37.554297    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:37.861883 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:37.861745    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"]
Nov 03 16:21:37.873025 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:37.872992    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"]
Nov 03 16:21:37.966191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:37.965093    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:38.121587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:38.121495    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:38.308889 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:38.308854    6506 generic.go:356] "Generic (PLEG): container finished" podID="854b1626-e3b6-43dc-82a2-d83512df2718" containerID="e6c2a3e3780191ca8989710c1eeae264827b5ececc471118eb27bb73a66b252c" exitCode=0
Nov 03 16:21:38.320745 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:38.309639    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j" event={"ID":"854b1626-e3b6-43dc-82a2-d83512df2718","Type":"ContainerDied","Data":"e6c2a3e3780191ca8989710c1eeae264827b5ececc471118eb27bb73a66b252c"}
Nov 03 16:21:38.588682 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:38.588654    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"
Nov 03 16:21:38.680744 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:38.680714    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4deb4ddf-30da-4eb3-b56a-237274761339-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4deb4ddf-30da-4eb3-b56a-237274761339-maas-api-openshift-ca\") pod \"4deb4ddf-30da-4eb3-b56a-237274761339\" (UID: \"4deb4ddf-30da-4eb3-b56a-237274761339\") "
Nov 03 16:21:38.680954 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:38.680882    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4deb4ddf-30da-4eb3-b56a-237274761339-kube-api-access-gwtj8\" (UniqueName: \"kubernetes.io/projected/4deb4ddf-30da-4eb3-b56a-237274761339-kube-api-access-gwtj8\") pod \"4deb4ddf-30da-4eb3-b56a-237274761339\" (UID: \"4deb4ddf-30da-4eb3-b56a-237274761339\") "
Nov 03 16:21:38.681133 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:38.681110    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4deb4ddf-30da-4eb3-b56a-237274761339-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4deb4ddf-30da-4eb3-b56a-237274761339-maas-api-xks-ca\") pod \"4deb4ddf-30da-4eb3-b56a-237274761339\" (UID: \"4deb4ddf-30da-4eb3-b56a-237274761339\") "
Nov 03 16:21:38.681197 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:38.681152    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4deb4ddf-30da-4eb3-b56a-237274761339-maas-api-openshift-ca" pod "4deb4ddf-30da-4eb3-b56a-237274761339" (UID: "4deb4ddf-30da-4eb3-b56a-237274761339"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:38.681620 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:38.681600    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4deb4ddf-30da-4eb3-b56a-237274761339-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:38.683091 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:38.683068    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4deb4ddf-30da-4eb3-b56a-237274761339-kube-api-access-gwtj8" pod "4deb4ddf-30da-4eb3-b56a-237274761339" (UID: "4deb4ddf-30da-4eb3-b56a-237274761339"). InnerVolumeSpecName "kube-api-access-gwtj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:38.683192 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:38.683156    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4deb4ddf-30da-4eb3-b56a-237274761339-maas-api-xks-ca" pod "4deb4ddf-30da-4eb3-b56a-237274761339" (UID: "4deb4ddf-30da-4eb3-b56a-237274761339"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:38.783054 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:38.783025    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwtj8\" (UniqueName: \"kubernetes.io/projected/4deb4ddf-30da-4eb3-b56a-237274761339-kube-api-access-gwtj8\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:38.783054 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:38.783053    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4deb4ddf-30da-4eb3-b56a-237274761339-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:39.052831 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.052692    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:39.265825 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.265775    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"]
Nov 03 16:21:39.266474 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.266454    6506 cpu_manager.go:414] "containerMap: removing container" podUID="4deb4ddf-30da-4eb3-b56a-237274761339" containerName="revoke-keys"
Nov 03 16:21:39.266474 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.266473    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4deb4ddf-30da-4eb3-b56a-237274761339" containerName="revoke-keys"
Nov 03 16:21:39.266632 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.266610    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="4deb4ddf-30da-4eb3-b56a-237274761339" containerName="revoke-keys"
Nov 03 16:21:39.276321 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.276278    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.319855 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.319766    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"
Nov 03 16:21:39.320083 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.320031    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj" event={"ID":"4deb4ddf-30da-4eb3-b56a-237274761339","Type":"ContainerDied","Data":"c3d8e580f4608310387d7078d17bffd3040f25fff8f9d61a11ebc550d4a3c251"}
Nov 03 16:21:39.320170 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.320103    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3d8e580f4608310387d7078d17bffd3040f25fff8f9d61a11ebc550d4a3c251"
Nov 03 16:21:39.393258 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.393212    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.393442 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.393371    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.393442 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.393415    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24bd837-1361-49af-bb52-dd51f01967a7-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.393557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.393497    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.393557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.393541    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.393671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.393567    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24bd837-1361-49af-bb52-dd51f01967a7-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.393671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.393593    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.393671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.393646    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb6sw\" (UniqueName: \"kubernetes.io/projected/b24bd837-1361-49af-bb52-dd51f01967a7-kube-api-access-mb6sw\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.393942 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.393717    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24bd837-1361-49af-bb52-dd51f01967a7-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.439366 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.439334    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"]
Nov 03 16:21:39.495057 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.495013    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.495254 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.495069    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24bd837-1361-49af-bb52-dd51f01967a7-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.495254 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.495097    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.495254 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.495156    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb6sw\" (UniqueName: \"kubernetes.io/projected/b24bd837-1361-49af-bb52-dd51f01967a7-kube-api-access-mb6sw\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.495254 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.495231    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24bd837-1361-49af-bb52-dd51f01967a7-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.495491 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.495335    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.495491 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.495477    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.495604 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.495515    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24bd837-1361-49af-bb52-dd51f01967a7-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.495657 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.495607    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.495760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.495653    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.496106 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.496079    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.496220 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.496198    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.496443 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.496417    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.496585 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.496561    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24bd837-1361-49af-bb52-dd51f01967a7-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.502587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.502555    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24bd837-1361-49af-bb52-dd51f01967a7-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.504696 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.504669    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.515557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.515516    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24bd837-1361-49af-bb52-dd51f01967a7-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.518216 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.518196    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb6sw\" (UniqueName: \"kubernetes.io/projected/b24bd837-1361-49af-bb52-dd51f01967a7-kube-api-access-mb6sw\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.528708 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.528690    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"
Nov 03 16:21:39.608388 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.604485    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:39.608388 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.604979    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/854b1626-e3b6-43dc-82a2-d83512df2718-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/854b1626-e3b6-43dc-82a2-d83512df2718-maas-api-openshift-ca\") pod \"854b1626-e3b6-43dc-82a2-d83512df2718\" (UID: \"854b1626-e3b6-43dc-82a2-d83512df2718\") "
Nov 03 16:21:39.608388 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.605178    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/854b1626-e3b6-43dc-82a2-d83512df2718-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/854b1626-e3b6-43dc-82a2-d83512df2718-maas-api-xks-ca\") pod \"854b1626-e3b6-43dc-82a2-d83512df2718\" (UID: \"854b1626-e3b6-43dc-82a2-d83512df2718\") "
Nov 03 16:21:39.608388 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.605229    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854b1626-e3b6-43dc-82a2-d83512df2718-kube-api-access-lgl26\" (UniqueName: \"kubernetes.io/projected/854b1626-e3b6-43dc-82a2-d83512df2718-kube-api-access-lgl26\") pod \"854b1626-e3b6-43dc-82a2-d83512df2718\" (UID: \"854b1626-e3b6-43dc-82a2-d83512df2718\") "
Nov 03 16:21:39.608388 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.606252    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/854b1626-e3b6-43dc-82a2-d83512df2718-maas-api-openshift-ca" pod "854b1626-e3b6-43dc-82a2-d83512df2718" (UID: "854b1626-e3b6-43dc-82a2-d83512df2718"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:39.623417 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.623369    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854b1626-e3b6-43dc-82a2-d83512df2718-kube-api-access-lgl26" pod "854b1626-e3b6-43dc-82a2-d83512df2718" (UID: "854b1626-e3b6-43dc-82a2-d83512df2718"). InnerVolumeSpecName "kube-api-access-lgl26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:39.630753 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.629934    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/854b1626-e3b6-43dc-82a2-d83512df2718-maas-api-xks-ca" pod "854b1626-e3b6-43dc-82a2-d83512df2718" (UID: "854b1626-e3b6-43dc-82a2-d83512df2718"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:39.683545 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.683500    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"]
Nov 03 16:21:39.709252 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.709220    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/854b1626-e3b6-43dc-82a2-d83512df2718-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:39.709358 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.709256    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/854b1626-e3b6-43dc-82a2-d83512df2718-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:39.709358 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.709274    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgl26\" (UniqueName: \"kubernetes.io/projected/854b1626-e3b6-43dc-82a2-d83512df2718-kube-api-access-lgl26\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:39.898075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.897992    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:39.962894 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:21:39.962858    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24bd837_1361_49af_bb52_dd51f01967a7.slice/crio-34f0365d4a741227a671dab7d49c42ad6af4e9793e0b36619f9d38e36a61c951 WatchSource:0}: Error finding container 34f0365d4a741227a671dab7d49c42ad6af4e9793e0b36619f9d38e36a61c951: Status 404 returned error can't find the container with id 34f0365d4a741227a671dab7d49c42ad6af4e9793e0b36619f9d38e36a61c951
Nov 03 16:21:39.965748 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.965714    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:21:39.965880 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.965809    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"]
Nov 03 16:21:39.965880 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.965817    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:21:39.965880 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:39.965866    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:21:40.360928 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:40.360892    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" event={"ID":"b24bd837-1361-49af-bb52-dd51f01967a7","Type":"ContainerStarted","Data":"94c0489fc4c6f194c5baf122536b4b4d41c49209483c84bdcb86b15aa4959887"}
Nov 03 16:21:40.361924 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:40.361898    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" event={"ID":"b24bd837-1361-49af-bb52-dd51f01967a7","Type":"ContainerStarted","Data":"34f0365d4a741227a671dab7d49c42ad6af4e9793e0b36619f9d38e36a61c951"}
Nov 03 16:21:40.363481 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:40.363457    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j" event={"ID":"854b1626-e3b6-43dc-82a2-d83512df2718","Type":"ContainerDied","Data":"1cc753a03b381a5abdfaa5c47a4cc2383604140d3f6f113490c4bffc4fc2593e"}
Nov 03 16:21:40.363569 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:40.363498    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"
Nov 03 16:21:40.363840 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:40.363502    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cc753a03b381a5abdfaa5c47a4cc2383604140d3f6f113490c4bffc4fc2593e"
Nov 03 16:21:40.605599 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:40.605566    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:40.700614 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:40.700519    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:40.951883 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:40.950235    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:41.505823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:41.494518    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:41.684137 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:41.684022    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:21:41.684137 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:41.684085    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 16:21:41.692872 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:41.684845    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:42.400957 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:42.400908    6506 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:42.457997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:42.457949    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 16:21:42.502107 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:42.502064    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:42.607934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:42.607898    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:21:42.608522 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:42.608493    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 16:21:42.624426 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:42.624390    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:42.632531 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:42.632493    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:42.907614 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:42.907572    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n" podUID="dde0ec03-70f8-47bb-af69-6a56524d425f" containerName="payload-processing" containerID="cri-o://c2ea69746fae98fd8d8694fa50e08db9670abda8c0717fd6beb5e3588399bb33" gracePeriod=25
Nov 03 16:21:42.908850 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:42.908817    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:42.944189 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:42.944009    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g" podUID="80ee90cc-3ecf-40be-9d34-511fb0b42721" containerName="payload-pre-processing" containerID="cri-o://feebba58ac9d4615c0bb7af2125418be4b4f25dd8aa62b7a925a437c7d372fc2" gracePeriod=25
Nov 03 16:21:42.973045 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:42.973006    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt" podUID="a78c2dad-d700-4697-9220-c08a07a3d4c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:21:43.165594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:43.164939    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:43.420034 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:43.419935    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:43.461929 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:43.461880    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:43.502985 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:43.479467    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:21:43.548821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:43.545525    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:43.554820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:43.549488    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:21:43.605412 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:43.605293    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:21:43.605412 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:43.605347    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 16:21:43.713910 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:43.685147    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 16:21:43.713910 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:43.685598    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:21:43.713910 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:43.687122    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podStartSLOduration=4.687103701 podStartE2EDuration="4.687103701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:40.4372676 +0000 UTC m=+1114.450665764" watchObservedRunningTime="2026-11-03 16:21:43.687103701 +0000 UTC m=+1117.700501848"
Nov 03 16:21:43.713910 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:43.703993    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Nov 03 16:21:43.713910 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:43.704434    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:21:43.941369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:43.941314    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:43.954907 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:43.954864    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:21:44.182950 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:44.182902    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:44.440878 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:44.430621    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:44.475837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:44.475117    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:21:44.475837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:44.475182    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:21:44.519823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:44.519106    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 16:21:44.648846 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:44.641144    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:21:44.652974 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:44.652935    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 16:21:44.669294 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:44.669247    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:44.692998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:44.692900    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:44.706978 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:44.704436    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:45.016880 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:45.010107    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 16:21:45.209839 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:45.208959    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:45.507450 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:45.477977    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:21:45.523823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:45.520042    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 16:21:45.688160 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:45.688112    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:45.698988 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:45.688810    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:21:45.714072 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:45.714025    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 16:21:45.714731 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:45.712321    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 16:21:45.793823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:45.712426    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:45.922627 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:45.922591    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:46.010822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:46.009008    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:46.027890 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:46.027824    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:46.224868 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:46.224830    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:21:46.515105 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:46.515057    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:46.533175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:46.533129    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:46.694969 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:46.694931    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:21:46.730711 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:46.730655    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 16:21:47.057196 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:46.716337    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 16:21:47.182822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:46.886081    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 16:21:47.205865 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.009615    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:47.206670 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.026854    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:47.206963 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.029202    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:47.213205 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:21:47.168646    6506 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.067s"
Nov 03 16:21:47.221896 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.221863    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"]
Nov 03 16:21:47.222181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.222162    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"]
Nov 03 16:21:47.229815 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.222489    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj" podUID="c01949d1-06c5-4f23-855d-6e3b04d0abe8" containerName="istio-proxy" containerID="cri-o://c85d52c611d3516814cd30257a4c8fbc6d697d216808fe4ae6b851b34d652a35" gracePeriod=30
Nov 03 16:21:47.264030 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.263991    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms" podUID="05348976-46ce-43dc-9cd3-a7500709956a" containerName="istio-proxy" containerID="cri-o://6e41cb498685f4060b6e5015ff149d992633e310ad1bf2d35f444a1c0e4ab787" gracePeriod=30
Nov 03 16:21:47.349857 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.323319    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:47.361144 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.360998    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:21:47.361144 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.361062    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs" podUID="063fc995-de52-43cf-a70f-0f2d163e015f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:47.460823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.421582    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"]
Nov 03 16:21:47.460823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.421836    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx" podUID="9066cf9a-5d56-4a7d-b60b-cc881a2a0138" containerName="istio-proxy" containerID="cri-o://d5b16f7aa6a4d29f81f88813f5a3012ee9a326af227e152d011963332aebefde" gracePeriod=30
Nov 03 16:21:47.510958 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.510920    6506 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2m69r" podUID="b017bcd4-4c3b-489d-9f98-3d8840edda8e" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.25:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:47.518834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.517487    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:47.560714 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.534413    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:47.560714 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.540550    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:47.584882 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.570651    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:21:47.584882 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.570708    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w" podUID="39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:47.610359 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.610323    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:21:47.649453 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.649418    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 16:21:47.654379 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.611075    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt" event={"ID":"a78c2dad-d700-4697-9220-c08a07a3d4c3","Type":"ContainerDied","Data":"aeeff5f3f449f200493af9d617a18fe75e0fbb61fb59a2fb8eba8ee68de0a604"}
Nov 03 16:21:47.679932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.611046    6506 generic.go:356] "Generic (PLEG): container finished" podID="a78c2dad-d700-4697-9220-c08a07a3d4c3" containerID="aeeff5f3f449f200493af9d617a18fe75e0fbb61fb59a2fb8eba8ee68de0a604" exitCode=0
Nov 03 16:21:47.782914 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.782883    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"]
Nov 03 16:21:47.953680 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:21:47.953575    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78c2dad_d700_4697_9220_c08a07a3d4c3.slice/crio-aeeff5f3f449f200493af9d617a18fe75e0fbb61fb59a2fb8eba8ee68de0a604.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78c2dad_d700_4697_9220_c08a07a3d4c3.slice/crio-conmon-aeeff5f3f449f200493af9d617a18fe75e0fbb61fb59a2fb8eba8ee68de0a604.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:21:47.984703 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:47.984654    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" containerID="cri-o://c391304efb49bbf7c56702121140abef8afc97273b21c393454fca9a78e5b81d" gracePeriod=45
Nov 03 16:21:48.196356 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:48.184386    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:48.260841 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:48.223945    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:48.306942 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:48.306898    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:48.333424 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:48.333381    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 16:21:48.419823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:48.339437    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": dial tcp 10.133.0.74:8000: connect: connection refused"
Nov 03 16:21:48.577820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:48.421172    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:21:48.600060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:48.518149    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:21:48.607181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:48.518194    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:48.607181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:48.535826    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:48.607181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:48.600314    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx" podUID="9066cf9a-5d56-4a7d-b60b-cc881a2a0138" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:21:48.831111 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:48.831015    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:21:48.845906 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:48.845865    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 16:21:48.880342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:48.841556    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": dial tcp 10.133.0.74:8000: connect: connection refused"
Nov 03 16:21:48.902857 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:48.899652    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:49.223944 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:49.222390    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:49.294940 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:49.293561    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:49.411166 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:49.386918    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 16:21:49.535854 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:49.495443    6506 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:21:49.535854 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:49.495523    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj" podUID="c01949d1-06c5-4f23-855d-6e3b04d0abe8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:21:49.632989 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:49.632938    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:21:49.633248 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:49.633218    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 16:21:49.659848 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:49.634212    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:49.659848 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:49.641915    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:49.659848 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:49.641971    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:49.754868 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:49.737634    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": dial tcp 10.133.0.74:8000: connect: connection refused"
Nov 03 16:21:49.763349 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:49.753340    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"
Nov 03 16:21:49.763737 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:49.754063    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:21:50.053348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.053298    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:21:50.061815 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.059869    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main" containerID="cri-o://93c2d3d4511299f8eed5be3b3ac8b0c41595de2d13abadf704e2d5e783d3812d" gracePeriod=45
Nov 03 16:21:50.061815 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.061290    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:21:50.061815 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.061361    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:21:50.061815 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.061405    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:21:50.068132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.062837    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a78c2dad-d700-4697-9220-c08a07a3d4c3-kube-api-access-9mstc\" (UniqueName: \"kubernetes.io/projected/a78c2dad-d700-4697-9220-c08a07a3d4c3-kube-api-access-9mstc\") pod \"a78c2dad-d700-4697-9220-c08a07a3d4c3\" (UID: \"a78c2dad-d700-4697-9220-c08a07a3d4c3\") "
Nov 03 16:21:50.068132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.062911    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a78c2dad-d700-4697-9220-c08a07a3d4c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78c2dad-d700-4697-9220-c08a07a3d4c3-plugins-config-volume\") pod \"a78c2dad-d700-4697-9220-c08a07a3d4c3\" (UID: \"a78c2dad-d700-4697-9220-c08a07a3d4c3\") "
Nov 03 16:21:50.068132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.064075    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a78c2dad-d700-4697-9220-c08a07a3d4c3-plugins-config-volume" pod "a78c2dad-d700-4697-9220-c08a07a3d4c3" (UID: "a78c2dad-d700-4697-9220-c08a07a3d4c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:50.068132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.066321    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:21:50.076422 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.076145    6506 generic.go:356] "Generic (PLEG): container finished" podID="65518370-5977-4201-9caa-029d28cf964d" containerID="c391304efb49bbf7c56702121140abef8afc97273b21c393454fca9a78e5b81d" exitCode=0
Nov 03 16:21:50.076422 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.076248    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" event={"ID":"65518370-5977-4201-9caa-029d28cf964d","Type":"ContainerDied","Data":"c391304efb49bbf7c56702121140abef8afc97273b21c393454fca9a78e5b81d"}
Nov 03 16:21:50.076422 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.076368    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78c2dad-d700-4697-9220-c08a07a3d4c3-kube-api-access-9mstc" pod "a78c2dad-d700-4697-9220-c08a07a3d4c3" (UID: "a78c2dad-d700-4697-9220-c08a07a3d4c3"). InnerVolumeSpecName "kube-api-access-9mstc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:50.082645 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.080277    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt" event={"ID":"a78c2dad-d700-4697-9220-c08a07a3d4c3","Type":"ContainerDied","Data":"521c9997d9758b05db6aaec0ffac0750b7e1da4518a04268d8654df9d15638b8"}
Nov 03 16:21:50.082645 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.080330    6506 scope.go:122] "RemoveContainer" containerID="aeeff5f3f449f200493af9d617a18fe75e0fbb61fb59a2fb8eba8ee68de0a604"
Nov 03 16:21:50.082645 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.080461    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"
Nov 03 16:21:50.087122 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.086462    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main" containerID="cri-o://b23f829d209c30f065c601132de685b661f89401a15cf2008feac8ded822f575" gracePeriod=45
Nov 03 16:21:50.170962 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.169699    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mstc\" (UniqueName: \"kubernetes.io/projected/a78c2dad-d700-4697-9220-c08a07a3d4c3-kube-api-access-9mstc\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.170962 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.169733    6506 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78c2dad-d700-4697-9220-c08a07a3d4c3-plugins-config-volume\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.289285 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.289021    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"]
Nov 03 16:21:50.296821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.296751    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d569fb-696d5f4fdc-5tckt"]
Nov 03 16:21:50.358925 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.358521    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:21:50.382920 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.380596    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-home\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-home\") pod \"65518370-5977-4201-9caa-029d28cf964d\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") "
Nov 03 16:21:50.382920 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.380652    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65518370-5977-4201-9caa-029d28cf964d-tls-certs\" (UniqueName: \"kubernetes.io/secret/65518370-5977-4201-9caa-029d28cf964d-tls-certs\") pod \"65518370-5977-4201-9caa-029d28cf964d\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") "
Nov 03 16:21:50.382920 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.380816    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65518370-5977-4201-9caa-029d28cf964d-kube-api-access-hjvmn\" (UniqueName: \"kubernetes.io/projected/65518370-5977-4201-9caa-029d28cf964d-kube-api-access-hjvmn\") pod \"65518370-5977-4201-9caa-029d28cf964d\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") "
Nov 03 16:21:50.382920 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.381355    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-model-cache\") pod \"65518370-5977-4201-9caa-029d28cf964d\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") "
Nov 03 16:21:50.382920 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.381443    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-dshm\") pod \"65518370-5977-4201-9caa-029d28cf964d\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") "
Nov 03 16:21:50.382920 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.381573    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-tmp-dir\") pod \"65518370-5977-4201-9caa-029d28cf964d\" (UID: \"65518370-5977-4201-9caa-029d28cf964d\") "
Nov 03 16:21:50.382920 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.382480    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-tmp-dir" pod "65518370-5977-4201-9caa-029d28cf964d" (UID: "65518370-5977-4201-9caa-029d28cf964d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:50.382920 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.382664    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-model-cache" pod "65518370-5977-4201-9caa-029d28cf964d" (UID: "65518370-5977-4201-9caa-029d28cf964d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:50.388887 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.388036    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-home" pod "65518370-5977-4201-9caa-029d28cf964d" (UID: "65518370-5977-4201-9caa-029d28cf964d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:50.389739 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.389703    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-dshm" pod "65518370-5977-4201-9caa-029d28cf964d" (UID: "65518370-5977-4201-9caa-029d28cf964d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:50.391175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.391140    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65518370-5977-4201-9caa-029d28cf964d-tls-certs" pod "65518370-5977-4201-9caa-029d28cf964d" (UID: "65518370-5977-4201-9caa-029d28cf964d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:50.391582 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.391550    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65518370-5977-4201-9caa-029d28cf964d-kube-api-access-hjvmn" pod "65518370-5977-4201-9caa-029d28cf964d" (UID: "65518370-5977-4201-9caa-029d28cf964d"). InnerVolumeSpecName "kube-api-access-hjvmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:50.485139 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.485105    6506 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-tmp-dir\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.485322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.485309    6506 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-home\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.485409 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.485399    6506 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65518370-5977-4201-9caa-029d28cf964d-tls-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.485490 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.485478    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjvmn\" (UniqueName: \"kubernetes.io/projected/65518370-5977-4201-9caa-029d28cf964d-kube-api-access-hjvmn\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.485576 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.485566    6506 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-model-cache\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.485650 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.485640    6506 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65518370-5977-4201-9caa-029d28cf964d-dshm\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.535297 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.534953    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:21:50.581906 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.581602    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:21:50.587070 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.587043    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cf9062b-2e57-4875-a717-a89d753afa28-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf9062b-2e57-4875-a717-a89d753afa28-tls-certs\") pod \"7cf9062b-2e57-4875-a717-a89d753afa28\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") "
Nov 03 16:21:50.587207 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.587160    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-home\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-home\") pod \"7cf9062b-2e57-4875-a717-a89d753afa28\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") "
Nov 03 16:21:50.587207 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.587199    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-tmp-dir\") pod \"7cf9062b-2e57-4875-a717-a89d753afa28\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") "
Nov 03 16:21:50.587543 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.587511    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-home" pod "7cf9062b-2e57-4875-a717-a89d753afa28" (UID: "7cf9062b-2e57-4875-a717-a89d753afa28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:50.587840 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.587814    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-tmp-dir" pod "7cf9062b-2e57-4875-a717-a89d753afa28" (UID: "7cf9062b-2e57-4875-a717-a89d753afa28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:50.588576 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.588547    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-dshm\") pod \"7cf9062b-2e57-4875-a717-a89d753afa28\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") "
Nov 03 16:21:50.588680 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.588664    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf9062b-2e57-4875-a717-a89d753afa28-kube-api-access-h9fzz\" (UniqueName: \"kubernetes.io/projected/7cf9062b-2e57-4875-a717-a89d753afa28-kube-api-access-h9fzz\") pod \"7cf9062b-2e57-4875-a717-a89d753afa28\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") "
Nov 03 16:21:50.588740 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.588727    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-model-cache\") pod \"7cf9062b-2e57-4875-a717-a89d753afa28\" (UID: \"7cf9062b-2e57-4875-a717-a89d753afa28\") "
Nov 03 16:21:50.589494 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.589475    6506 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-home\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.589633 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.589619    6506 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-tmp-dir\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.591170 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.591127    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-model-cache" pod "7cf9062b-2e57-4875-a717-a89d753afa28" (UID: "7cf9062b-2e57-4875-a717-a89d753afa28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:50.592689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.592624    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf9062b-2e57-4875-a717-a89d753afa28-kube-api-access-h9fzz" pod "7cf9062b-2e57-4875-a717-a89d753afa28" (UID: "7cf9062b-2e57-4875-a717-a89d753afa28"). InnerVolumeSpecName "kube-api-access-h9fzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:50.595156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.595123    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-dshm" pod "7cf9062b-2e57-4875-a717-a89d753afa28" (UID: "7cf9062b-2e57-4875-a717-a89d753afa28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:50.609889 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.606034    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:21:50.609889 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.606088    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 16:21:50.630265 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.630216    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf9062b-2e57-4875-a717-a89d753afa28-tls-certs" pod "7cf9062b-2e57-4875-a717-a89d753afa28" (UID: "7cf9062b-2e57-4875-a717-a89d753afa28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:50.691152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.691109    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-home\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-home\") pod \"9d30c691-0b1a-45e5-948b-8826c1173d37\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") "
Nov 03 16:21:50.691756 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.691699    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-home" pod "9d30c691-0b1a-45e5-948b-8826c1173d37" (UID: "9d30c691-0b1a-45e5-948b-8826c1173d37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:50.692853 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.692013    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d30c691-0b1a-45e5-948b-8826c1173d37-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d30c691-0b1a-45e5-948b-8826c1173d37-tls-certs\") pod \"9d30c691-0b1a-45e5-948b-8826c1173d37\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") "
Nov 03 16:21:50.692853 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.692109    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-model-cache\") pod \"9d30c691-0b1a-45e5-948b-8826c1173d37\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") "
Nov 03 16:21:50.692853 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.692172    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-dshm\") pod \"9d30c691-0b1a-45e5-948b-8826c1173d37\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") "
Nov 03 16:21:50.692853 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.692215    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d30c691-0b1a-45e5-948b-8826c1173d37-kube-api-access-dw8wg\" (UniqueName: \"kubernetes.io/projected/9d30c691-0b1a-45e5-948b-8826c1173d37-kube-api-access-dw8wg\") pod \"9d30c691-0b1a-45e5-948b-8826c1173d37\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") "
Nov 03 16:21:50.692853 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.692266    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-tmp-dir\") pod \"9d30c691-0b1a-45e5-948b-8826c1173d37\" (UID: \"9d30c691-0b1a-45e5-948b-8826c1173d37\") "
Nov 03 16:21:50.694262 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.694004    6506 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-dshm\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.694262 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.694029    6506 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-home\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.694262 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.694046    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9fzz\" (UniqueName: \"kubernetes.io/projected/7cf9062b-2e57-4875-a717-a89d753afa28-kube-api-access-h9fzz\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.694262 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.694060    6506 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cf9062b-2e57-4875-a717-a89d753afa28-model-cache\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.694262 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.694073    6506 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf9062b-2e57-4875-a717-a89d753afa28-tls-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.695754 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.695638    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-model-cache" pod "9d30c691-0b1a-45e5-948b-8826c1173d37" (UID: "9d30c691-0b1a-45e5-948b-8826c1173d37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:50.696714 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.696557    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-tmp-dir" pod "9d30c691-0b1a-45e5-948b-8826c1173d37" (UID: "9d30c691-0b1a-45e5-948b-8826c1173d37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:50.699775 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.699754    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-dshm" pod "9d30c691-0b1a-45e5-948b-8826c1173d37" (UID: "9d30c691-0b1a-45e5-948b-8826c1173d37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:50.700131 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.700098    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d30c691-0b1a-45e5-948b-8826c1173d37-kube-api-access-dw8wg" pod "9d30c691-0b1a-45e5-948b-8826c1173d37" (UID: "9d30c691-0b1a-45e5-948b-8826c1173d37"). InnerVolumeSpecName "kube-api-access-dw8wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:50.700871 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.700844    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d30c691-0b1a-45e5-948b-8826c1173d37-tls-certs" pod "9d30c691-0b1a-45e5-948b-8826c1173d37" (UID: "9d30c691-0b1a-45e5-948b-8826c1173d37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:50.795870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.795839    6506 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d30c691-0b1a-45e5-948b-8826c1173d37-tls-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.795870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.795870    6506 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-model-cache\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.796065 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.795880    6506 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-dshm\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.796065 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.795890    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw8wg\" (UniqueName: \"kubernetes.io/projected/9d30c691-0b1a-45e5-948b-8826c1173d37-kube-api-access-dw8wg\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.796065 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:50.795900    6506 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d30c691-0b1a-45e5-948b-8826c1173d37-tmp-dir\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:51.160953 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.152253    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv" event={"ID":"65518370-5977-4201-9caa-029d28cf964d","Type":"ContainerDied","Data":"62153db20d98f19d1a8d4bb45b701f77ab4d771140452255c6d4e8a9aa360d70"}
Nov 03 16:21:51.160953 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.152313    6506 scope.go:122] "RemoveContainer" containerID="c391304efb49bbf7c56702121140abef8afc97273b21c393454fca9a78e5b81d"
Nov 03 16:21:51.160953 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.152431    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"
Nov 03 16:21:51.170058 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.169992    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:51.182372 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.182156    6506 generic.go:356] "Generic (PLEG): container finished" podID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerID="93c2d3d4511299f8eed5be3b3ac8b0c41595de2d13abadf704e2d5e783d3812d" exitCode=0
Nov 03 16:21:51.182762 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.182704    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"
Nov 03 16:21:51.183054 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.182921    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" event={"ID":"9d30c691-0b1a-45e5-948b-8826c1173d37","Type":"ContainerDied","Data":"93c2d3d4511299f8eed5be3b3ac8b0c41595de2d13abadf704e2d5e783d3812d"}
Nov 03 16:21:51.183054 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.182972    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc" event={"ID":"9d30c691-0b1a-45e5-948b-8826c1173d37","Type":"ContainerDied","Data":"31ce980e922788144ce1a35d8beee93ca6e2c27af9b5889d52739dc880f56f51"}
Nov 03 16:21:51.197287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.193964    6506 generic.go:356] "Generic (PLEG): container finished" podID="7cf9062b-2e57-4875-a717-a89d753afa28" containerID="b23f829d209c30f065c601132de685b661f89401a15cf2008feac8ded822f575" exitCode=0
Nov 03 16:21:51.197287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.194093    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" event={"ID":"7cf9062b-2e57-4875-a717-a89d753afa28","Type":"ContainerDied","Data":"b23f829d209c30f065c601132de685b661f89401a15cf2008feac8ded822f575"}
Nov 03 16:21:51.197287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.194132    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf" event={"ID":"7cf9062b-2e57-4875-a717-a89d753afa28","Type":"ContainerDied","Data":"e979ef782f11a5711c07f23307dcf90f239264e71a8548b7b0683233491b8018"}
Nov 03 16:21:51.197287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.194197    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"
Nov 03 16:21:51.197287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.196944    6506 scope.go:122] "RemoveContainer" containerID="93c2d3d4511299f8eed5be3b3ac8b0c41595de2d13abadf704e2d5e783d3812d"
Nov 03 16:21:51.245243 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.244898    6506 scope.go:122] "RemoveContainer" containerID="93c2d3d4511299f8eed5be3b3ac8b0c41595de2d13abadf704e2d5e783d3812d"
Nov 03 16:21:51.245470 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:21:51.245447    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c2d3d4511299f8eed5be3b3ac8b0c41595de2d13abadf704e2d5e783d3812d\": container with ID starting with 93c2d3d4511299f8eed5be3b3ac8b0c41595de2d13abadf704e2d5e783d3812d not found: ID does not exist" containerID="93c2d3d4511299f8eed5be3b3ac8b0c41595de2d13abadf704e2d5e783d3812d"
Nov 03 16:21:51.246103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.245485    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c2d3d4511299f8eed5be3b3ac8b0c41595de2d13abadf704e2d5e783d3812d"} err="failed to get container status \"93c2d3d4511299f8eed5be3b3ac8b0c41595de2d13abadf704e2d5e783d3812d\": rpc error: code = NotFound desc = could not find container \"93c2d3d4511299f8eed5be3b3ac8b0c41595de2d13abadf704e2d5e783d3812d\": container with ID starting with 93c2d3d4511299f8eed5be3b3ac8b0c41595de2d13abadf704e2d5e783d3812d not found: ID does not exist"
Nov 03 16:21:51.246103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.245513    6506 scope.go:122] "RemoveContainer" containerID="b23f829d209c30f065c601132de685b661f89401a15cf2008feac8ded822f575"
Nov 03 16:21:51.253389 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.253336    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"]
Nov 03 16:21:51.258995 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.258971    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/distinct-w1-c09b17-kserve-6fc76765cb-dh8xv"]
Nov 03 16:21:51.259747 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.259734    6506 scope.go:122] "RemoveContainer" containerID="b23f829d209c30f065c601132de685b661f89401a15cf2008feac8ded822f575"
Nov 03 16:21:51.260047 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:21:51.260023    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b23f829d209c30f065c601132de685b661f89401a15cf2008feac8ded822f575\": container with ID starting with b23f829d209c30f065c601132de685b661f89401a15cf2008feac8ded822f575 not found: ID does not exist" containerID="b23f829d209c30f065c601132de685b661f89401a15cf2008feac8ded822f575"
Nov 03 16:21:51.260110 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.260055    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b23f829d209c30f065c601132de685b661f89401a15cf2008feac8ded822f575"} err="failed to get container status \"b23f829d209c30f065c601132de685b661f89401a15cf2008feac8ded822f575\": rpc error: code = NotFound desc = could not find container \"b23f829d209c30f065c601132de685b661f89401a15cf2008feac8ded822f575\": container with ID starting with b23f829d209c30f065c601132de685b661f89401a15cf2008feac8ded822f575 not found: ID does not exist"
Nov 03 16:21:51.264120 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.264098    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"]
Nov 03 16:21:51.267900 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.267877    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/simulator-w6-afb990-kserve-5f658f4968-bl5jf"]
Nov 03 16:21:51.284869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.284840    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"]
Nov 03 16:21:51.288757 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.288733    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/premium-w6-afb990-kserve-84c945bdfc-r97wc"]
Nov 03 16:21:51.356230 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.356148    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:51.605741 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.605708    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:21:51.605922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.605757    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 16:21:51.878075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:51.878037    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g" podUID="80ee90cc-3ecf-40be-9d34-511fb0b42721" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:21:52.053900 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.053858    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"]
Nov 03 16:21:52.055135 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.055095    6506 cpu_manager.go:414] "containerMap: removing container" podUID="a78c2dad-d700-4697-9220-c08a07a3d4c3" containerName="payload-pre-processing"
Nov 03 16:21:52.055135 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.055125    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78c2dad-d700-4697-9220-c08a07a3d4c3" containerName="payload-pre-processing"
Nov 03 16:21:52.055301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.055167    6506 cpu_manager.go:414] "containerMap: removing container" podUID="854b1626-e3b6-43dc-82a2-d83512df2718" containerName="revoke-keys"
Nov 03 16:21:52.055301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.055176    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854b1626-e3b6-43dc-82a2-d83512df2718" containerName="revoke-keys"
Nov 03 16:21:52.055301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.055222    6506 cpu_manager.go:414] "containerMap: removing container" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main"
Nov 03 16:21:52.055301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.055230    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main"
Nov 03 16:21:52.055301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.055243    6506 cpu_manager.go:414] "containerMap: removing container" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main"
Nov 03 16:21:52.055301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.055251    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main"
Nov 03 16:21:52.055301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.055280    6506 cpu_manager.go:414] "containerMap: removing container" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main"
Nov 03 16:21:52.055301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.055289    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main"
Nov 03 16:21:52.055537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.055443    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="854b1626-e3b6-43dc-82a2-d83512df2718" containerName="revoke-keys"
Nov 03 16:21:52.055537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.055461    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" containerName="main"
Nov 03 16:21:52.055537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.055470    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" containerName="main"
Nov 03 16:21:52.055537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.055482    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="65518370-5977-4201-9caa-029d28cf964d" containerName="main"
Nov 03 16:21:52.055537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.055491    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78c2dad-d700-4697-9220-c08a07a3d4c3" containerName="payload-pre-processing"
Nov 03 16:21:52.067510 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.067482    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"]
Nov 03 16:21:52.067647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.067607    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"
Nov 03 16:21:52.072706 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.072682    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:21:52.072882 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.072731    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:21:52.137511 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.107677    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65518370-5977-4201-9caa-029d28cf964d" path="/var/lib/kubelet/pods/65518370-5977-4201-9caa-029d28cf964d/volumes"
Nov 03 16:21:52.138596 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.138554    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cf9062b-2e57-4875-a717-a89d753afa28" path="/var/lib/kubelet/pods/7cf9062b-2e57-4875-a717-a89d753afa28/volumes"
Nov 03 16:21:52.139164 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.139133    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d30c691-0b1a-45e5-948b-8826c1173d37" path="/var/lib/kubelet/pods/9d30c691-0b1a-45e5-948b-8826c1173d37/volumes"
Nov 03 16:21:52.139612 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.139583    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a78c2dad-d700-4697-9220-c08a07a3d4c3" path="/var/lib/kubelet/pods/a78c2dad-d700-4697-9220-c08a07a3d4c3/volumes"
Nov 03 16:21:52.163662 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.163640    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpxzz\" (UniqueName: \"kubernetes.io/projected/bcc0055d-6da5-4866-915c-b81476bfecbd-kube-api-access-wpxzz\") pod \"maas-api-revoke-sub-108034f4-gl9h8\" (UID: \"bcc0055d-6da5-4866-915c-b81476bfecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"
Nov 03 16:21:52.163836 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.163702    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc0055d-6da5-4866-915c-b81476bfecbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-108034f4-gl9h8\" (UID: \"bcc0055d-6da5-4866-915c-b81476bfecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"
Nov 03 16:21:52.163980 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.163960    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc0055d-6da5-4866-915c-b81476bfecbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-108034f4-gl9h8\" (UID: \"bcc0055d-6da5-4866-915c-b81476bfecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"
Nov 03 16:21:52.266805 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.266131    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc0055d-6da5-4866-915c-b81476bfecbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-108034f4-gl9h8\" (UID: \"bcc0055d-6da5-4866-915c-b81476bfecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"
Nov 03 16:21:52.269511 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.269485    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc0055d-6da5-4866-915c-b81476bfecbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-108034f4-gl9h8\" (UID: \"bcc0055d-6da5-4866-915c-b81476bfecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"
Nov 03 16:21:52.269726 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.269602    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpxzz\" (UniqueName: \"kubernetes.io/projected/bcc0055d-6da5-4866-915c-b81476bfecbd-kube-api-access-wpxzz\") pod \"maas-api-revoke-sub-108034f4-gl9h8\" (UID: \"bcc0055d-6da5-4866-915c-b81476bfecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"
Nov 03 16:21:52.276956 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.276927    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc0055d-6da5-4866-915c-b81476bfecbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-108034f4-gl9h8\" (UID: \"bcc0055d-6da5-4866-915c-b81476bfecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"
Nov 03 16:21:52.280083 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.280035    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc0055d-6da5-4866-915c-b81476bfecbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-108034f4-gl9h8\" (UID: \"bcc0055d-6da5-4866-915c-b81476bfecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"
Nov 03 16:21:52.289265 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.289236    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpxzz\" (UniqueName: \"kubernetes.io/projected/bcc0055d-6da5-4866-915c-b81476bfecbd-kube-api-access-wpxzz\") pod \"maas-api-revoke-sub-108034f4-gl9h8\" (UID: \"bcc0055d-6da5-4866-915c-b81476bfecbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"
Nov 03 16:21:52.389481 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.389404    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:52.391513 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.391492    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"
Nov 03 16:21:52.626822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.611872    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:21:52.626822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.611958    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 16:21:52.775363 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.774911    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"]
Nov 03 16:21:52.783545 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.783515    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c" podUID="381eb05d-c83e-49b3-89b3-935ea62fb4e1" containerName="payload-processing" containerID="cri-o://c86b1a39aff6e45af98ab2633d29df483eec7f5621650ed9df88268e0f8d7efd" gracePeriod=25
Nov 03 16:21:52.938850 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.938462    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n" podUID="dde0ec03-70f8-47bb-af69-6a56524d425f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:21:52.995579 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:52.995278    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:53.016887 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.016859    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:21:53.091610 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.091571    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-envoy\") pod \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") "
Nov 03 16:21:53.091822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.091725    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-podinfo\") pod \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") "
Nov 03 16:21:53.091822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.091760    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-credential-socket\") pod \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") "
Nov 03 16:21:53.091941 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.091866    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-workload-socket\") pod \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") "
Nov 03 16:21:53.091941 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.091928    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-token\" (UniqueName: \"kubernetes.io/projected/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-token\") pod \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") "
Nov 03 16:21:53.092083 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.091993    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istiod-ca-cert\") pod \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") "
Nov 03 16:21:53.092083 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.092048    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-data\") pod \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") "
Nov 03 16:21:53.092199 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.092143    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01949d1-06c5-4f23-855d-6e3b04d0abe8-kube-api-access-hwtmr\" (UniqueName: \"kubernetes.io/projected/c01949d1-06c5-4f23-855d-6e3b04d0abe8-kube-api-access-hwtmr\") pod \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") "
Nov 03 16:21:53.092199 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.092171    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-workload-certs\") pod \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\" (UID: \"c01949d1-06c5-4f23-855d-6e3b04d0abe8\") "
Nov 03 16:21:53.092296 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.092205    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-workload-socket" pod "c01949d1-06c5-4f23-855d-6e3b04d0abe8" (UID: "c01949d1-06c5-4f23-855d-6e3b04d0abe8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.092564 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.092537    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-data" pod "c01949d1-06c5-4f23-855d-6e3b04d0abe8" (UID: "c01949d1-06c5-4f23-855d-6e3b04d0abe8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.092847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.092820    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-workload-certs" pod "c01949d1-06c5-4f23-855d-6e3b04d0abe8" (UID: "c01949d1-06c5-4f23-855d-6e3b04d0abe8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.092940 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.092916    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istiod-ca-cert" pod "c01949d1-06c5-4f23-855d-6e3b04d0abe8" (UID: "c01949d1-06c5-4f23-855d-6e3b04d0abe8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:53.093315 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.093220    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.093315 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.093247    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.093315 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.093263    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.093315 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.093277    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.093315 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.093220    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-credential-socket" pod "c01949d1-06c5-4f23-855d-6e3b04d0abe8" (UID: "c01949d1-06c5-4f23-855d-6e3b04d0abe8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.108649 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.108612    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01949d1-06c5-4f23-855d-6e3b04d0abe8-kube-api-access-hwtmr" pod "c01949d1-06c5-4f23-855d-6e3b04d0abe8" (UID: "c01949d1-06c5-4f23-855d-6e3b04d0abe8"). InnerVolumeSpecName "kube-api-access-hwtmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:53.108869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.108664    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-token" pod "c01949d1-06c5-4f23-855d-6e3b04d0abe8" (UID: "c01949d1-06c5-4f23-855d-6e3b04d0abe8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:53.108869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.108675    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-podinfo" pod "c01949d1-06c5-4f23-855d-6e3b04d0abe8" (UID: "c01949d1-06c5-4f23-855d-6e3b04d0abe8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:21:53.109131 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.109088    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-envoy" pod "c01949d1-06c5-4f23-855d-6e3b04d0abe8" (UID: "c01949d1-06c5-4f23-855d-6e3b04d0abe8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.133993 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.133964    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:53.194294 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.194225    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-workload-socket\") pod \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") "
Nov 03 16:21:53.194294 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.194256    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-workload-certs\") pod \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") "
Nov 03 16:21:53.194294 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.194287    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-credential-socket\") pod \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") "
Nov 03 16:21:53.194517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.194315    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istiod-ca-cert\") pod \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") "
Nov 03 16:21:53.194517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.194385    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-envoy\") pod \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") "
Nov 03 16:21:53.194517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.194435    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-token\" (UniqueName: \"kubernetes.io/projected/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-token\") pod \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") "
Nov 03 16:21:53.194517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.194452    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-data\") pod \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") "
Nov 03 16:21:53.194517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.194498    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-workload-socket" pod "9066cf9a-5d56-4a7d-b60b-cc881a2a0138" (UID: "9066cf9a-5d56-4a7d-b60b-cc881a2a0138"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.194778 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.194555    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-podinfo\") pod \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") "
Nov 03 16:21:53.194778 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.194597    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-kube-api-access-vkgn6\" (UniqueName: \"kubernetes.io/projected/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-kube-api-access-vkgn6\") pod \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\" (UID: \"9066cf9a-5d56-4a7d-b60b-cc881a2a0138\") "
Nov 03 16:21:53.194778 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.194603    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-workload-certs" pod "9066cf9a-5d56-4a7d-b60b-cc881a2a0138" (UID: "9066cf9a-5d56-4a7d-b60b-cc881a2a0138"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.194778 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.194679    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-credential-socket" pod "9066cf9a-5d56-4a7d-b60b-cc881a2a0138" (UID: "9066cf9a-5d56-4a7d-b60b-cc881a2a0138"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.194778 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.194754    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-data" pod "9066cf9a-5d56-4a7d-b60b-cc881a2a0138" (UID: "9066cf9a-5d56-4a7d-b60b-cc881a2a0138"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.194989 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.194917    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istiod-ca-cert" pod "9066cf9a-5d56-4a7d-b60b-cc881a2a0138" (UID: "9066cf9a-5d56-4a7d-b60b-cc881a2a0138"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:53.195633 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.195308    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.195633 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.195333    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.195633 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.195349    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.195633 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.195362    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.195633 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.195378    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.195633 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.195392    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.195633 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.195408    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.195633 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.195422    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.195633 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.195437    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwtmr\" (UniqueName: \"kubernetes.io/projected/c01949d1-06c5-4f23-855d-6e3b04d0abe8-kube-api-access-hwtmr\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.195633 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.195488    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01949d1-06c5-4f23-855d-6e3b04d0abe8-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.196981 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.196954    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-envoy" pod "9066cf9a-5d56-4a7d-b60b-cc881a2a0138" (UID: "9066cf9a-5d56-4a7d-b60b-cc881a2a0138"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.196981 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.196959    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-kube-api-access-vkgn6" pod "9066cf9a-5d56-4a7d-b60b-cc881a2a0138" (UID: "9066cf9a-5d56-4a7d-b60b-cc881a2a0138"). InnerVolumeSpecName "kube-api-access-vkgn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:53.197235 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.197220    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-podinfo" pod "9066cf9a-5d56-4a7d-b60b-cc881a2a0138" (UID: "9066cf9a-5d56-4a7d-b60b-cc881a2a0138"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:21:53.197235 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.197221    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-token" pod "9066cf9a-5d56-4a7d-b60b-cc881a2a0138" (UID: "9066cf9a-5d56-4a7d-b60b-cc881a2a0138"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:53.214361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.214333    6506 generic.go:356] "Generic (PLEG): container finished" podID="c01949d1-06c5-4f23-855d-6e3b04d0abe8" containerID="c85d52c611d3516814cd30257a4c8fbc6d697d216808fe4ae6b851b34d652a35" exitCode=0
Nov 03 16:21:53.214492 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.214384    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"
Nov 03 16:21:53.214492 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.214413    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj" event={"ID":"c01949d1-06c5-4f23-855d-6e3b04d0abe8","Type":"ContainerDied","Data":"c85d52c611d3516814cd30257a4c8fbc6d697d216808fe4ae6b851b34d652a35"}
Nov 03 16:21:53.214492 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.214468    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj" event={"ID":"c01949d1-06c5-4f23-855d-6e3b04d0abe8","Type":"ContainerDied","Data":"f49cae83479927cf443707d35dcaaf0d296a9621fbaddb1bd3913ba87591df70"}
Nov 03 16:21:53.214652 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.214497    6506 scope.go:122] "RemoveContainer" containerID="c85d52c611d3516814cd30257a4c8fbc6d697d216808fe4ae6b851b34d652a35"
Nov 03 16:21:53.216085 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.216057    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8" event={"ID":"bcc0055d-6da5-4866-915c-b81476bfecbd","Type":"ContainerStarted","Data":"59524c80ca88cdfdaefa2dbc9397d6ddd2f7a6b0fe136c9a967eb44ec961868b"}
Nov 03 16:21:53.217898 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.217876    6506 generic.go:356] "Generic (PLEG): container finished" podID="05348976-46ce-43dc-9cd3-a7500709956a" containerID="6e41cb498685f4060b6e5015ff149d992633e310ad1bf2d35f444a1c0e4ab787" exitCode=0
Nov 03 16:21:53.217981 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.217954    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms" event={"ID":"05348976-46ce-43dc-9cd3-a7500709956a","Type":"ContainerDied","Data":"6e41cb498685f4060b6e5015ff149d992633e310ad1bf2d35f444a1c0e4ab787"}
Nov 03 16:21:53.230122 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.220980    6506 generic.go:356] "Generic (PLEG): container finished" podID="9066cf9a-5d56-4a7d-b60b-cc881a2a0138" containerID="d5b16f7aa6a4d29f81f88813f5a3012ee9a326af227e152d011963332aebefde" exitCode=0
Nov 03 16:21:53.230122 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.221027    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx" event={"ID":"9066cf9a-5d56-4a7d-b60b-cc881a2a0138","Type":"ContainerDied","Data":"d5b16f7aa6a4d29f81f88813f5a3012ee9a326af227e152d011963332aebefde"}
Nov 03 16:21:53.230122 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.221064    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx" event={"ID":"9066cf9a-5d56-4a7d-b60b-cc881a2a0138","Type":"ContainerDied","Data":"9da26764a91d5dc202a8641323b68c7962f65ff0a36fb99c32ecdeef140423c9"}
Nov 03 16:21:53.230122 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.221144    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"
Nov 03 16:21:53.230122 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.228548    6506 scope.go:122] "RemoveContainer" containerID="c85d52c611d3516814cd30257a4c8fbc6d697d216808fe4ae6b851b34d652a35"
Nov 03 16:21:53.230122 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:21:53.228869    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c85d52c611d3516814cd30257a4c8fbc6d697d216808fe4ae6b851b34d652a35\": container with ID starting with c85d52c611d3516814cd30257a4c8fbc6d697d216808fe4ae6b851b34d652a35 not found: ID does not exist" containerID="c85d52c611d3516814cd30257a4c8fbc6d697d216808fe4ae6b851b34d652a35"
Nov 03 16:21:53.230122 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.228889    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85d52c611d3516814cd30257a4c8fbc6d697d216808fe4ae6b851b34d652a35"} err="failed to get container status \"c85d52c611d3516814cd30257a4c8fbc6d697d216808fe4ae6b851b34d652a35\": rpc error: code = NotFound desc = could not find container \"c85d52c611d3516814cd30257a4c8fbc6d697d216808fe4ae6b851b34d652a35\": container with ID starting with c85d52c611d3516814cd30257a4c8fbc6d697d216808fe4ae6b851b34d652a35 not found: ID does not exist"
Nov 03 16:21:53.230122 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.228907    6506 scope.go:122] "RemoveContainer" containerID="d5b16f7aa6a4d29f81f88813f5a3012ee9a326af227e152d011963332aebefde"
Nov 03 16:21:53.238910 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.238894    6506 scope.go:122] "RemoveContainer" containerID="d5b16f7aa6a4d29f81f88813f5a3012ee9a326af227e152d011963332aebefde"
Nov 03 16:21:53.239202 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:21:53.239184    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b16f7aa6a4d29f81f88813f5a3012ee9a326af227e152d011963332aebefde\": container with ID starting with d5b16f7aa6a4d29f81f88813f5a3012ee9a326af227e152d011963332aebefde not found: ID does not exist" containerID="d5b16f7aa6a4d29f81f88813f5a3012ee9a326af227e152d011963332aebefde"
Nov 03 16:21:53.239257 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.239209    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b16f7aa6a4d29f81f88813f5a3012ee9a326af227e152d011963332aebefde"} err="failed to get container status \"d5b16f7aa6a4d29f81f88813f5a3012ee9a326af227e152d011963332aebefde\": rpc error: code = NotFound desc = could not find container \"d5b16f7aa6a4d29f81f88813f5a3012ee9a326af227e152d011963332aebefde\": container with ID starting with d5b16f7aa6a4d29f81f88813f5a3012ee9a326af227e152d011963332aebefde not found: ID does not exist"
Nov 03 16:21:53.249161 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.249138    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"]
Nov 03 16:21:53.255873 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.255850    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-v4nbj"]
Nov 03 16:21:53.272397 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.272373    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"]
Nov 03 16:21:53.278660 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.278635    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-pd2mx"]
Nov 03 16:21:53.297215 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.297194    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.297215 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.297215    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.297337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.297226    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.297337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.297234    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkgn6\" (UniqueName: \"kubernetes.io/projected/9066cf9a-5d56-4a7d-b60b-cc881a2a0138-kube-api-access-vkgn6\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.505650 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.505550    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:53.574684 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.574654    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:21:53.605474 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.605435    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:21:53.606236 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.606198    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 16:21:53.649349 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.649300    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"]
Nov 03 16:21:53.653650 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.653623    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7196a-th5ts"]
Nov 03 16:21:53.704337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.704309    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05348976-46ce-43dc-9cd3-a7500709956a-istio-token\" (UniqueName: \"kubernetes.io/projected/05348976-46ce-43dc-9cd3-a7500709956a-istio-token\") pod \"05348976-46ce-43dc-9cd3-a7500709956a\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") "
Nov 03 16:21:53.704489 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.704347    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-workload-certs\") pod \"05348976-46ce-43dc-9cd3-a7500709956a\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") "
Nov 03 16:21:53.704489 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.704384    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-istio-data\") pod \"05348976-46ce-43dc-9cd3-a7500709956a\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") "
Nov 03 16:21:53.704597 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.704528    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05348976-46ce-43dc-9cd3-a7500709956a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05348976-46ce-43dc-9cd3-a7500709956a-istiod-ca-cert\") pod \"05348976-46ce-43dc-9cd3-a7500709956a\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") "
Nov 03 16:21:53.704597 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.704583    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-workload-socket\") pod \"05348976-46ce-43dc-9cd3-a7500709956a\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") "
Nov 03 16:21:53.704726 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.704610    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-istio-envoy\") pod \"05348976-46ce-43dc-9cd3-a7500709956a\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") "
Nov 03 16:21:53.704726 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.704684    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-credential-socket\") pod \"05348976-46ce-43dc-9cd3-a7500709956a\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") "
Nov 03 16:21:53.704726 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.704720    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05348976-46ce-43dc-9cd3-a7500709956a-kube-api-access-c9g6f\" (UniqueName: \"kubernetes.io/projected/05348976-46ce-43dc-9cd3-a7500709956a-kube-api-access-c9g6f\") pod \"05348976-46ce-43dc-9cd3-a7500709956a\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") "
Nov 03 16:21:53.704901 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.704724    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-istio-data" pod "05348976-46ce-43dc-9cd3-a7500709956a" (UID: "05348976-46ce-43dc-9cd3-a7500709956a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.704901 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.704766    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05348976-46ce-43dc-9cd3-a7500709956a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05348976-46ce-43dc-9cd3-a7500709956a-istio-podinfo\") pod \"05348976-46ce-43dc-9cd3-a7500709956a\" (UID: \"05348976-46ce-43dc-9cd3-a7500709956a\") "
Nov 03 16:21:53.704901 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.704715    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-workload-certs" pod "05348976-46ce-43dc-9cd3-a7500709956a" (UID: "05348976-46ce-43dc-9cd3-a7500709956a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.705045 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.704978    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-workload-socket" pod "05348976-46ce-43dc-9cd3-a7500709956a" (UID: "05348976-46ce-43dc-9cd3-a7500709956a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.705045 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.705017    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05348976-46ce-43dc-9cd3-a7500709956a-istiod-ca-cert" pod "05348976-46ce-43dc-9cd3-a7500709956a" (UID: "05348976-46ce-43dc-9cd3-a7500709956a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:53.705045 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.705025    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-credential-socket" pod "05348976-46ce-43dc-9cd3-a7500709956a" (UID: "05348976-46ce-43dc-9cd3-a7500709956a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.705488 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.705463    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.705569 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.705494    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.705569 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.705509    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.705569 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.705525    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05348976-46ce-43dc-9cd3-a7500709956a-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.705569 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.705539    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.706936 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.706901    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05348976-46ce-43dc-9cd3-a7500709956a-istio-token" pod "05348976-46ce-43dc-9cd3-a7500709956a" (UID: "05348976-46ce-43dc-9cd3-a7500709956a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:53.707496 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.707471    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-istio-envoy" pod "05348976-46ce-43dc-9cd3-a7500709956a" (UID: "05348976-46ce-43dc-9cd3-a7500709956a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.707590 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.707515    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05348976-46ce-43dc-9cd3-a7500709956a-kube-api-access-c9g6f" pod "05348976-46ce-43dc-9cd3-a7500709956a" (UID: "05348976-46ce-43dc-9cd3-a7500709956a"). InnerVolumeSpecName "kube-api-access-c9g6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:53.707651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.707592    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05348976-46ce-43dc-9cd3-a7500709956a-istio-podinfo" pod "05348976-46ce-43dc-9cd3-a7500709956a" (UID: "05348976-46ce-43dc-9cd3-a7500709956a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:21:53.770426 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.770398    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:53.807201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.807176    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05348976-46ce-43dc-9cd3-a7500709956a-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.807337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.807215    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9g6f\" (UniqueName: \"kubernetes.io/projected/05348976-46ce-43dc-9cd3-a7500709956a-kube-api-access-c9g6f\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.807337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.807230    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05348976-46ce-43dc-9cd3-a7500709956a-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.807337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:53.807243    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05348976-46ce-43dc-9cd3-a7500709956a-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.109059 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:54.109022    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115bede2-738c-48f7-af23-efd9bf52c022" path="/var/lib/kubelet/pods/115bede2-738c-48f7-af23-efd9bf52c022/volumes"
Nov 03 16:21:54.109783 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:54.109760    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9066cf9a-5d56-4a7d-b60b-cc881a2a0138" path="/var/lib/kubelet/pods/9066cf9a-5d56-4a7d-b60b-cc881a2a0138/volumes"
Nov 03 16:21:54.110545 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:54.110512    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c01949d1-06c5-4f23-855d-6e3b04d0abe8" path="/var/lib/kubelet/pods/c01949d1-06c5-4f23-855d-6e3b04d0abe8/volumes"
Nov 03 16:21:54.151307 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:54.151262    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:54.234854 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:54.234778    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms" event={"ID":"05348976-46ce-43dc-9cd3-a7500709956a","Type":"ContainerDied","Data":"80bd736887967badf044140730e40cc3d9db49418558bb9ee72b63e1ed44ab14"}
Nov 03 16:21:54.235056 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:54.234826    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"
Nov 03 16:21:54.235056 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:54.234934    6506 scope.go:122] "RemoveContainer" containerID="6e41cb498685f4060b6e5015ff149d992633e310ad1bf2d35f444a1c0e4ab787"
Nov 03 16:21:54.240202 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:54.240174    6506 generic.go:356] "Generic (PLEG): container finished" podID="bcc0055d-6da5-4866-915c-b81476bfecbd" containerID="c17d1e150c08262e3e01fbf864b297d069c7e6b28103a458d6fbc86a4f80fed5" exitCode=0
Nov 03 16:21:54.240296 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:54.240227    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8" event={"ID":"bcc0055d-6da5-4866-915c-b81476bfecbd","Type":"ContainerDied","Data":"c17d1e150c08262e3e01fbf864b297d069c7e6b28103a458d6fbc86a4f80fed5"}
Nov 03 16:21:54.261460 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:54.261435    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"]
Nov 03 16:21:54.266345 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:54.266317    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-dqwms"]
Nov 03 16:21:54.602375 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:54.602292    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:54.605191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:54.605172    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:54.609599 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:54.609579    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:54.719580 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:54.719538    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main" containerID="cri-o://3c37186f351e58ff9984590f1350e79ad87fa9210b852e2cca3300ead48ee58d" gracePeriod=45
Nov 03 16:21:55.027199 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.027173    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:21:55.125699 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.125669    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-home\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-home\") pod \"f4867d2d-c38c-44d6-924c-b0afe704f841\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") "
Nov 03 16:21:55.125945 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.125928    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-tmp-dir\") pod \"f4867d2d-c38c-44d6-924c-b0afe704f841\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") "
Nov 03 16:21:55.126058 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.125927    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-home" pod "f4867d2d-c38c-44d6-924c-b0afe704f841" (UID: "f4867d2d-c38c-44d6-924c-b0afe704f841"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.126126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.126040    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4867d2d-c38c-44d6-924c-b0afe704f841-kube-api-access-bsrfc\" (UniqueName: \"kubernetes.io/projected/f4867d2d-c38c-44d6-924c-b0afe704f841-kube-api-access-bsrfc\") pod \"f4867d2d-c38c-44d6-924c-b0afe704f841\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") "
Nov 03 16:21:55.126126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.126119    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4867d2d-c38c-44d6-924c-b0afe704f841-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4867d2d-c38c-44d6-924c-b0afe704f841-tls-certs\") pod \"f4867d2d-c38c-44d6-924c-b0afe704f841\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") "
Nov 03 16:21:55.126240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.126035    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-tmp-dir" pod "f4867d2d-c38c-44d6-924c-b0afe704f841" (UID: "f4867d2d-c38c-44d6-924c-b0afe704f841"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.126240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.126173    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-dshm\") pod \"f4867d2d-c38c-44d6-924c-b0afe704f841\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") "
Nov 03 16:21:55.126240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.126204    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-model-cache\") pod \"f4867d2d-c38c-44d6-924c-b0afe704f841\" (UID: \"f4867d2d-c38c-44d6-924c-b0afe704f841\") "
Nov 03 16:21:55.126524 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.126502    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-model-cache" pod "f4867d2d-c38c-44d6-924c-b0afe704f841" (UID: "f4867d2d-c38c-44d6-924c-b0afe704f841"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.126750 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.126721    6506 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-tmp-dir\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.126750 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.126745    6506 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-model-cache\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.126944 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.126761    6506 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-home\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.128299 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.128272    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4867d2d-c38c-44d6-924c-b0afe704f841-tls-certs" pod "f4867d2d-c38c-44d6-924c-b0afe704f841" (UID: "f4867d2d-c38c-44d6-924c-b0afe704f841"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:55.128392 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.128322    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4867d2d-c38c-44d6-924c-b0afe704f841-kube-api-access-bsrfc" pod "f4867d2d-c38c-44d6-924c-b0afe704f841" (UID: "f4867d2d-c38c-44d6-924c-b0afe704f841"). InnerVolumeSpecName "kube-api-access-bsrfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:55.128392 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.128362    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-dshm" pod "f4867d2d-c38c-44d6-924c-b0afe704f841" (UID: "f4867d2d-c38c-44d6-924c-b0afe704f841"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.228496 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.228419    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsrfc\" (UniqueName: \"kubernetes.io/projected/f4867d2d-c38c-44d6-924c-b0afe704f841-kube-api-access-bsrfc\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.228496 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.228458    6506 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4867d2d-c38c-44d6-924c-b0afe704f841-tls-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.228496 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.228473    6506 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4867d2d-c38c-44d6-924c-b0afe704f841-dshm\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.249412 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.249385    6506 generic.go:356] "Generic (PLEG): container finished" podID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerID="3c37186f351e58ff9984590f1350e79ad87fa9210b852e2cca3300ead48ee58d" exitCode=0
Nov 03 16:21:55.249566 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.249442    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"
Nov 03 16:21:55.249566 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.249472    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" event={"ID":"f4867d2d-c38c-44d6-924c-b0afe704f841","Type":"ContainerDied","Data":"3c37186f351e58ff9984590f1350e79ad87fa9210b852e2cca3300ead48ee58d"}
Nov 03 16:21:55.249566 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.249517    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n" event={"ID":"f4867d2d-c38c-44d6-924c-b0afe704f841","Type":"ContainerDied","Data":"62b15bb76137d23a27f0af9cda51125df6641e93ee2c6ca1552c831c44113e92"}
Nov 03 16:21:55.249566 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.249535    6506 scope.go:122] "RemoveContainer" containerID="3c37186f351e58ff9984590f1350e79ad87fa9210b852e2cca3300ead48ee58d"
Nov 03 16:21:55.252281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.252262    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:21:55.261220 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.261204    6506 scope.go:122] "RemoveContainer" containerID="3c37186f351e58ff9984590f1350e79ad87fa9210b852e2cca3300ead48ee58d"
Nov 03 16:21:55.261535 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:21:55.261517    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c37186f351e58ff9984590f1350e79ad87fa9210b852e2cca3300ead48ee58d\": container with ID starting with 3c37186f351e58ff9984590f1350e79ad87fa9210b852e2cca3300ead48ee58d not found: ID does not exist" containerID="3c37186f351e58ff9984590f1350e79ad87fa9210b852e2cca3300ead48ee58d"
Nov 03 16:21:55.261575 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.261543    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c37186f351e58ff9984590f1350e79ad87fa9210b852e2cca3300ead48ee58d"} err="failed to get container status \"3c37186f351e58ff9984590f1350e79ad87fa9210b852e2cca3300ead48ee58d\": rpc error: code = NotFound desc = could not find container \"3c37186f351e58ff9984590f1350e79ad87fa9210b852e2cca3300ead48ee58d\": container with ID starting with 3c37186f351e58ff9984590f1350e79ad87fa9210b852e2cca3300ead48ee58d not found: ID does not exist"
Nov 03 16:21:55.313745 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.313705    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"]
Nov 03 16:21:55.318598 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.318066    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/premium-w5-69cedd-kserve-68977c5d54-cv58n"]
Nov 03 16:21:55.449465 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.449441    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"
Nov 03 16:21:55.638588 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.638555    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc0055d-6da5-4866-915c-b81476bfecbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc0055d-6da5-4866-915c-b81476bfecbd-maas-api-openshift-ca\") pod \"bcc0055d-6da5-4866-915c-b81476bfecbd\" (UID: \"bcc0055d-6da5-4866-915c-b81476bfecbd\") "
Nov 03 16:21:55.639029 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.638622    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcc0055d-6da5-4866-915c-b81476bfecbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc0055d-6da5-4866-915c-b81476bfecbd-maas-api-xks-ca\") pod \"bcc0055d-6da5-4866-915c-b81476bfecbd\" (UID: \"bcc0055d-6da5-4866-915c-b81476bfecbd\") "
Nov 03 16:21:55.639029 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.638831    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc0055d-6da5-4866-915c-b81476bfecbd-kube-api-access-wpxzz\" (UniqueName: \"kubernetes.io/projected/bcc0055d-6da5-4866-915c-b81476bfecbd-kube-api-access-wpxzz\") pod \"bcc0055d-6da5-4866-915c-b81476bfecbd\" (UID: \"bcc0055d-6da5-4866-915c-b81476bfecbd\") "
Nov 03 16:21:55.639029 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.639018    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc0055d-6da5-4866-915c-b81476bfecbd-maas-api-openshift-ca" pod "bcc0055d-6da5-4866-915c-b81476bfecbd" (UID: "bcc0055d-6da5-4866-915c-b81476bfecbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:55.639377 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.639362    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc0055d-6da5-4866-915c-b81476bfecbd-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.640897 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.640877    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc0055d-6da5-4866-915c-b81476bfecbd-maas-api-xks-ca" pod "bcc0055d-6da5-4866-915c-b81476bfecbd" (UID: "bcc0055d-6da5-4866-915c-b81476bfecbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:55.640979 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.640958    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc0055d-6da5-4866-915c-b81476bfecbd-kube-api-access-wpxzz" pod "bcc0055d-6da5-4866-915c-b81476bfecbd" (UID: "bcc0055d-6da5-4866-915c-b81476bfecbd"). InnerVolumeSpecName "kube-api-access-wpxzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:55.740311 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.740283    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc0055d-6da5-4866-915c-b81476bfecbd-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.740311 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:55.740309    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpxzz\" (UniqueName: \"kubernetes.io/projected/bcc0055d-6da5-4866-915c-b81476bfecbd-kube-api-access-wpxzz\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.047544 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.047509    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"]
Nov 03 16:21:56.048377 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.048350    6506 cpu_manager.go:414] "containerMap: removing container" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main"
Nov 03 16:21:56.048377 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.048374    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main"
Nov 03 16:21:56.048535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.048397    6506 cpu_manager.go:414] "containerMap: removing container" podUID="05348976-46ce-43dc-9cd3-a7500709956a" containerName="istio-proxy"
Nov 03 16:21:56.048535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.048406    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05348976-46ce-43dc-9cd3-a7500709956a" containerName="istio-proxy"
Nov 03 16:21:56.048535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.048429    6506 cpu_manager.go:414] "containerMap: removing container" podUID="c01949d1-06c5-4f23-855d-6e3b04d0abe8" containerName="istio-proxy"
Nov 03 16:21:56.048535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.048437    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01949d1-06c5-4f23-855d-6e3b04d0abe8" containerName="istio-proxy"
Nov 03 16:21:56.048535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.048462    6506 cpu_manager.go:414] "containerMap: removing container" podUID="bcc0055d-6da5-4866-915c-b81476bfecbd" containerName="revoke-keys"
Nov 03 16:21:56.048535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.048469    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc0055d-6da5-4866-915c-b81476bfecbd" containerName="revoke-keys"
Nov 03 16:21:56.048535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.048505    6506 cpu_manager.go:414] "containerMap: removing container" podUID="9066cf9a-5d56-4a7d-b60b-cc881a2a0138" containerName="istio-proxy"
Nov 03 16:21:56.048535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.048511    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9066cf9a-5d56-4a7d-b60b-cc881a2a0138" containerName="istio-proxy"
Nov 03 16:21:56.048874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.048638    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" containerName="main"
Nov 03 16:21:56.048874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.048648    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="05348976-46ce-43dc-9cd3-a7500709956a" containerName="istio-proxy"
Nov 03 16:21:56.048874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.048657    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01949d1-06c5-4f23-855d-6e3b04d0abe8" containerName="istio-proxy"
Nov 03 16:21:56.048874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.048666    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc0055d-6da5-4866-915c-b81476bfecbd" containerName="revoke-keys"
Nov 03 16:21:56.048874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.048674    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="9066cf9a-5d56-4a7d-b60b-cc881a2a0138" containerName="istio-proxy"
Nov 03 16:21:56.052449 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.052430    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"
Nov 03 16:21:56.068832 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.068768    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"]
Nov 03 16:21:56.129875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.129842    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05348976-46ce-43dc-9cd3-a7500709956a" path="/var/lib/kubelet/pods/05348976-46ce-43dc-9cd3-a7500709956a/volumes"
Nov 03 16:21:56.130335 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.130322    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4867d2d-c38c-44d6-924c-b0afe704f841" path="/var/lib/kubelet/pods/f4867d2d-c38c-44d6-924c-b0afe704f841/volumes"
Nov 03 16:21:56.145881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.145851    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a4e8de-d68f-4726-8699-d452f998cc64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4576c79-mfpgz\" (UID: \"52a4e8de-d68f-4726-8699-d452f998cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"
Nov 03 16:21:56.146004 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.145986    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k67m2\" (UniqueName: \"kubernetes.io/projected/52a4e8de-d68f-4726-8699-d452f998cc64-kube-api-access-k67m2\") pod \"maas-api-revoke-sub-a4576c79-mfpgz\" (UID: \"52a4e8de-d68f-4726-8699-d452f998cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"
Nov 03 16:21:56.146128 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.146100    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a4e8de-d68f-4726-8699-d452f998cc64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4576c79-mfpgz\" (UID: \"52a4e8de-d68f-4726-8699-d452f998cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"
Nov 03 16:21:56.247384 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.247357    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k67m2\" (UniqueName: \"kubernetes.io/projected/52a4e8de-d68f-4726-8699-d452f998cc64-kube-api-access-k67m2\") pod \"maas-api-revoke-sub-a4576c79-mfpgz\" (UID: \"52a4e8de-d68f-4726-8699-d452f998cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"
Nov 03 16:21:56.247542 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.247411    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a4e8de-d68f-4726-8699-d452f998cc64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4576c79-mfpgz\" (UID: \"52a4e8de-d68f-4726-8699-d452f998cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"
Nov 03 16:21:56.247581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.247540    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a4e8de-d68f-4726-8699-d452f998cc64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4576c79-mfpgz\" (UID: \"52a4e8de-d68f-4726-8699-d452f998cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"
Nov 03 16:21:56.248208 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.248187    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a4e8de-d68f-4726-8699-d452f998cc64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4576c79-mfpgz\" (UID: \"52a4e8de-d68f-4726-8699-d452f998cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"
Nov 03 16:21:56.249926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.249908    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a4e8de-d68f-4726-8699-d452f998cc64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4576c79-mfpgz\" (UID: \"52a4e8de-d68f-4726-8699-d452f998cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"
Nov 03 16:21:56.260260 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.260232    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"
Nov 03 16:21:56.260390 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.260235    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8" event={"ID":"bcc0055d-6da5-4866-915c-b81476bfecbd","Type":"ContainerDied","Data":"59524c80ca88cdfdaefa2dbc9397d6ddd2f7a6b0fe136c9a967eb44ec961868b"}
Nov 03 16:21:56.260390 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.260354    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59524c80ca88cdfdaefa2dbc9397d6ddd2f7a6b0fe136c9a967eb44ec961868b"
Nov 03 16:21:56.260390 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.260380    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k67m2\" (UniqueName: \"kubernetes.io/projected/52a4e8de-d68f-4726-8699-d452f998cc64-kube-api-access-k67m2\") pod \"maas-api-revoke-sub-a4576c79-mfpgz\" (UID: \"52a4e8de-d68f-4726-8699-d452f998cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"
Nov 03 16:21:56.377013 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.376939    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"
Nov 03 16:21:56.664710 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:56.664638    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"]
Nov 03 16:21:56.667918 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:21:56.667889    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a4e8de_d68f_4726_8699_d452f998cc64.slice/crio-7ff3fd1156843a21e2d3b2cc1c63e51a6117d07e235f786ee9f70d09a59b578f WatchSource:0}: Error finding container 7ff3fd1156843a21e2d3b2cc1c63e51a6117d07e235f786ee9f70d09a59b578f: Status 404 returned error can't find the container with id 7ff3fd1156843a21e2d3b2cc1c63e51a6117d07e235f786ee9f70d09a59b578f
Nov 03 16:21:57.127716 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:57.127675    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:57.272691 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:57.272662    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz" event={"ID":"52a4e8de-d68f-4726-8699-d452f998cc64","Type":"ContainerStarted","Data":"83c6cb44d3a308f9f7e46f6f519f3cfc4eb9071e5843c3cca8426fc6db13f3ce"}
Nov 03 16:21:57.272891 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:57.272710    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz" event={"ID":"52a4e8de-d68f-4726-8699-d452f998cc64","Type":"ContainerStarted","Data":"7ff3fd1156843a21e2d3b2cc1c63e51a6117d07e235f786ee9f70d09a59b578f"}
Nov 03 16:21:57.308967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:57.308919    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz" podStartSLOduration=1.3089053 podStartE2EDuration="1.3089053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:57.305852897 +0000 UTC m=+1131.319251043" watchObservedRunningTime="2026-11-03 16:21:57.3089053 +0000 UTC m=+1131.322303448"
Nov 03 16:21:57.793767 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:57.793727    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"]
Nov 03 16:21:57.798030 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:57.798013    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"
Nov 03 16:21:57.830648 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:57.830619    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"]
Nov 03 16:21:57.973775 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:57.973699    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d7abf06-4jdz2\" (UID: \"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"
Nov 03 16:21:57.973775 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:57.973746    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d7abf06-4jdz2\" (UID: \"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"
Nov 03 16:21:57.973985 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:57.973900    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwwc\" (UniqueName: \"kubernetes.io/projected/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-kube-api-access-2dwwc\") pod \"maas-api-revoke-sub-5d7abf06-4jdz2\" (UID: \"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"
Nov 03 16:21:58.075056 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:58.075023    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d7abf06-4jdz2\" (UID: \"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"
Nov 03 16:21:58.075203 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:58.075069    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d7abf06-4jdz2\" (UID: \"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"
Nov 03 16:21:58.075203 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:58.075172    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwwc\" (UniqueName: \"kubernetes.io/projected/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-kube-api-access-2dwwc\") pod \"maas-api-revoke-sub-5d7abf06-4jdz2\" (UID: \"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"
Nov 03 16:21:58.075696 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:58.075671    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d7abf06-4jdz2\" (UID: \"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"
Nov 03 16:21:58.077505 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:58.077486    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d7abf06-4jdz2\" (UID: \"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"
Nov 03 16:21:58.088208 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:58.088181    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwwc\" (UniqueName: \"kubernetes.io/projected/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-kube-api-access-2dwwc\") pod \"maas-api-revoke-sub-5d7abf06-4jdz2\" (UID: \"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"
Nov 03 16:21:58.110900 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:58.110874    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"
Nov 03 16:21:58.289514 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:58.289477    6506 generic.go:356] "Generic (PLEG): container finished" podID="52a4e8de-d68f-4726-8699-d452f998cc64" containerID="83c6cb44d3a308f9f7e46f6f519f3cfc4eb9071e5843c3cca8426fc6db13f3ce" exitCode=0
Nov 03 16:21:58.289702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:58.289550    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz" event={"ID":"52a4e8de-d68f-4726-8699-d452f998cc64","Type":"ContainerDied","Data":"83c6cb44d3a308f9f7e46f6f519f3cfc4eb9071e5843c3cca8426fc6db13f3ce"}
Nov 03 16:21:58.415580 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:21:58.415549    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f7f17c_0ba8_4117_8b7c_3a4627f36bcc.slice/crio-f9b155f8c8e4cd3f1d7f0e2f51304f34866d01ef8662bb608b4152e75d138e6e WatchSource:0}: Error finding container f9b155f8c8e4cd3f1d7f0e2f51304f34866d01ef8662bb608b4152e75d138e6e: Status 404 returned error can't find the container with id f9b155f8c8e4cd3f1d7f0e2f51304f34866d01ef8662bb608b4152e75d138e6e
Nov 03 16:21:58.415717 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:58.415635    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"]
Nov 03 16:21:58.797064 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:21:58.797030    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:21:58.852032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:58.851970    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:21:59.299321 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:59.299269    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2" event={"ID":"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc","Type":"ContainerStarted","Data":"75d32a1a94debd0cdc1a39ab16e9c495e1a3eb9480d708b6bf3d11c5ca1220af"}
Nov 03 16:21:59.299321 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:59.299324    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2" event={"ID":"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc","Type":"ContainerStarted","Data":"f9b155f8c8e4cd3f1d7f0e2f51304f34866d01ef8662bb608b4152e75d138e6e"}
Nov 03 16:21:59.323402 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:59.323348    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2" podStartSLOduration=2.32333057 podStartE2EDuration="2.32333057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:59.322134677 +0000 UTC m=+1133.335532824" watchObservedRunningTime="2026-11-03 16:21:59.32333057 +0000 UTC m=+1133.336728718"
Nov 03 16:21:59.466724 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:59.466698    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"
Nov 03 16:21:59.499656 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:59.499627    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a4e8de-d68f-4726-8699-d452f998cc64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a4e8de-d68f-4726-8699-d452f998cc64-maas-api-xks-ca\") pod \"52a4e8de-d68f-4726-8699-d452f998cc64\" (UID: \"52a4e8de-d68f-4726-8699-d452f998cc64\") "
Nov 03 16:21:59.499847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:59.499734    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a4e8de-d68f-4726-8699-d452f998cc64-kube-api-access-k67m2\" (UniqueName: \"kubernetes.io/projected/52a4e8de-d68f-4726-8699-d452f998cc64-kube-api-access-k67m2\") pod \"52a4e8de-d68f-4726-8699-d452f998cc64\" (UID: \"52a4e8de-d68f-4726-8699-d452f998cc64\") "
Nov 03 16:21:59.499924 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:59.499911    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a4e8de-d68f-4726-8699-d452f998cc64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a4e8de-d68f-4726-8699-d452f998cc64-maas-api-openshift-ca\") pod \"52a4e8de-d68f-4726-8699-d452f998cc64\" (UID: \"52a4e8de-d68f-4726-8699-d452f998cc64\") "
Nov 03 16:21:59.501189 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:59.501157    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a4e8de-d68f-4726-8699-d452f998cc64-maas-api-openshift-ca" pod "52a4e8de-d68f-4726-8699-d452f998cc64" (UID: "52a4e8de-d68f-4726-8699-d452f998cc64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:59.501957 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:59.501934    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a4e8de-d68f-4726-8699-d452f998cc64-maas-api-xks-ca" pod "52a4e8de-d68f-4726-8699-d452f998cc64" (UID: "52a4e8de-d68f-4726-8699-d452f998cc64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:59.502302 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:59.502277    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a4e8de-d68f-4726-8699-d452f998cc64-kube-api-access-k67m2" pod "52a4e8de-d68f-4726-8699-d452f998cc64" (UID: "52a4e8de-d68f-4726-8699-d452f998cc64"). InnerVolumeSpecName "kube-api-access-k67m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:59.601936 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:59.601861    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a4e8de-d68f-4726-8699-d452f998cc64-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:59.601936 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:59.601889    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a4e8de-d68f-4726-8699-d452f998cc64-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:59.601936 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:21:59.601903    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k67m2\" (UniqueName: \"kubernetes.io/projected/52a4e8de-d68f-4726-8699-d452f998cc64-kube-api-access-k67m2\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:00.308226 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:00.308190    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"
Nov 03 16:22:00.308226 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:00.308200    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz" event={"ID":"52a4e8de-d68f-4726-8699-d452f998cc64","Type":"ContainerDied","Data":"7ff3fd1156843a21e2d3b2cc1c63e51a6117d07e235f786ee9f70d09a59b578f"}
Nov 03 16:22:00.308672 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:00.308253    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ff3fd1156843a21e2d3b2cc1c63e51a6117d07e235f786ee9f70d09a59b578f"
Nov 03 16:22:00.310177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:00.310146    6506 generic.go:356] "Generic (PLEG): container finished" podID="b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc" containerID="75d32a1a94debd0cdc1a39ab16e9c495e1a3eb9480d708b6bf3d11c5ca1220af" exitCode=0
Nov 03 16:22:00.310324 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:00.310222    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2" event={"ID":"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc","Type":"ContainerDied","Data":"75d32a1a94debd0cdc1a39ab16e9c495e1a3eb9480d708b6bf3d11c5ca1220af"}
Nov 03 16:22:00.436885 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:00.436753    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:01.515881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:01.515855    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"
Nov 03 16:22:01.532249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:01.532226    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-kube-api-access-2dwwc\" (UniqueName: \"kubernetes.io/projected/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-kube-api-access-2dwwc\") pod \"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc\" (UID: \"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc\") "
Nov 03 16:22:01.532391 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:01.532354    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-maas-api-openshift-ca\") pod \"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc\" (UID: \"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc\") "
Nov 03 16:22:01.532518 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:01.532496    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-maas-api-xks-ca\") pod \"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc\" (UID: \"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc\") "
Nov 03 16:22:01.532858 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:01.532833    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-maas-api-openshift-ca" pod "b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc" (UID: "b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:01.533059 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:01.533045    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:01.534447 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:01.534422    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-kube-api-access-2dwwc" pod "b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc" (UID: "b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc"). InnerVolumeSpecName "kube-api-access-2dwwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:01.534818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:01.534783    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-maas-api-xks-ca" pod "b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc" (UID: "b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:01.634721 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:01.634643    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:01.634721 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:01.634676    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dwwc\" (UniqueName: \"kubernetes.io/projected/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc-kube-api-access-2dwwc\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:01.643912 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:01.643882    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:01.875199 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:01.875157    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g" podUID="80ee90cc-3ecf-40be-9d34-511fb0b42721" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:22:01.875346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:01.875253    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c" podUID="381eb05d-c83e-49b3-89b3-935ea62fb4e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:22:01.958329 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:01.958224    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:02.126419 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:02.126382    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:02.249197 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:02.249090    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"]
Nov 03 16:22:02.249371 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:02.249341    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t" podUID="05788e58-6105-4a9d-812f-5fcfc45a820c" containerName="istio-proxy" containerID="cri-o://7e55e78fdc2a481312d16b2c3ff7739adcda4aa78cdc3be83a9dafa4f4835f53" gracePeriod=30
Nov 03 16:22:02.259724 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:02.259690    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"]
Nov 03 16:22:02.259975 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:02.259947    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg" podUID="c81a5fc9-15c0-491f-bb7a-939e4485f544" containerName="istio-proxy" containerID="cri-o://03d73ada7084ffd770703c631df0f207ae30072fab9a450fa4e19f6a156c425d" gracePeriod=30
Nov 03 16:22:02.334934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:02.334883    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"
Nov 03 16:22:02.334934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:02.334909    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2" event={"ID":"b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc","Type":"ContainerDied","Data":"f9b155f8c8e4cd3f1d7f0e2f51304f34866d01ef8662bb608b4152e75d138e6e"}
Nov 03 16:22:02.335147 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:02.334960    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9b155f8c8e4cd3f1d7f0e2f51304f34866d01ef8662bb608b4152e75d138e6e"
Nov 03 16:22:02.342296 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:02.342270    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:02.630584 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:02.630546    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:02.631001 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:02.630616    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t" podUID="05788e58-6105-4a9d-812f-5fcfc45a820c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:02.886500 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:02.886410    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n" podUID="dde0ec03-70f8-47bb-af69-6a56524d425f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:22:03.367567 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.367247    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"
Nov 03 16:22:03.381651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.381613    6506 generic.go:356] "Generic (PLEG): container finished" podID="dde0ec03-70f8-47bb-af69-6a56524d425f" containerID="c2ea69746fae98fd8d8694fa50e08db9670abda8c0717fd6beb5e3588399bb33" exitCode=0
Nov 03 16:22:03.382015 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.381995    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n" event={"ID":"dde0ec03-70f8-47bb-af69-6a56524d425f","Type":"ContainerDied","Data":"c2ea69746fae98fd8d8694fa50e08db9670abda8c0717fd6beb5e3588399bb33"}
Nov 03 16:22:03.384342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.384313    6506 generic.go:356] "Generic (PLEG): container finished" podID="80ee90cc-3ecf-40be-9d34-511fb0b42721" containerID="feebba58ac9d4615c0bb7af2125418be4b4f25dd8aa62b7a925a437c7d372fc2" exitCode=0
Nov 03 16:22:03.384467 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.384372    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"
Nov 03 16:22:03.384467 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.384381    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g" event={"ID":"80ee90cc-3ecf-40be-9d34-511fb0b42721","Type":"ContainerDied","Data":"feebba58ac9d4615c0bb7af2125418be4b4f25dd8aa62b7a925a437c7d372fc2"}
Nov 03 16:22:03.384467 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.384422    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g" event={"ID":"80ee90cc-3ecf-40be-9d34-511fb0b42721","Type":"ContainerDied","Data":"3ba05e7161afeea8fab860ffe86877747fdea940296070b399f71aff304bdd3e"}
Nov 03 16:22:03.384467 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.384450    6506 scope.go:122] "RemoveContainer" containerID="feebba58ac9d4615c0bb7af2125418be4b4f25dd8aa62b7a925a437c7d372fc2"
Nov 03 16:22:03.412373 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.412352    6506 scope.go:122] "RemoveContainer" containerID="feebba58ac9d4615c0bb7af2125418be4b4f25dd8aa62b7a925a437c7d372fc2"
Nov 03 16:22:03.414927 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:03.414858    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feebba58ac9d4615c0bb7af2125418be4b4f25dd8aa62b7a925a437c7d372fc2\": container with ID starting with feebba58ac9d4615c0bb7af2125418be4b4f25dd8aa62b7a925a437c7d372fc2 not found: ID does not exist" containerID="feebba58ac9d4615c0bb7af2125418be4b4f25dd8aa62b7a925a437c7d372fc2"
Nov 03 16:22:03.415092 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.414924    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feebba58ac9d4615c0bb7af2125418be4b4f25dd8aa62b7a925a437c7d372fc2"} err="failed to get container status \"feebba58ac9d4615c0bb7af2125418be4b4f25dd8aa62b7a925a437c7d372fc2\": rpc error: code = NotFound desc = could not find container \"feebba58ac9d4615c0bb7af2125418be4b4f25dd8aa62b7a925a437c7d372fc2\": container with ID starting with feebba58ac9d4615c0bb7af2125418be4b4f25dd8aa62b7a925a437c7d372fc2 not found: ID does not exist"
Nov 03 16:22:03.416590 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.416551    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"
Nov 03 16:22:03.502103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.502071    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ee90cc-3ecf-40be-9d34-511fb0b42721-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ee90cc-3ecf-40be-9d34-511fb0b42721-plugins-config-volume\") pod \"80ee90cc-3ecf-40be-9d34-511fb0b42721\" (UID: \"80ee90cc-3ecf-40be-9d34-511fb0b42721\") "
Nov 03 16:22:03.502301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.502128    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde0ec03-70f8-47bb-af69-6a56524d425f-kube-api-access-gqf59\" (UniqueName: \"kubernetes.io/projected/dde0ec03-70f8-47bb-af69-6a56524d425f-kube-api-access-gqf59\") pod \"dde0ec03-70f8-47bb-af69-6a56524d425f\" (UID: \"dde0ec03-70f8-47bb-af69-6a56524d425f\") "
Nov 03 16:22:03.502301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.502168    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dde0ec03-70f8-47bb-af69-6a56524d425f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde0ec03-70f8-47bb-af69-6a56524d425f-plugins-config-volume\") pod \"dde0ec03-70f8-47bb-af69-6a56524d425f\" (UID: \"dde0ec03-70f8-47bb-af69-6a56524d425f\") "
Nov 03 16:22:03.502407 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.502335    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ee90cc-3ecf-40be-9d34-511fb0b42721-kube-api-access-fgntb\" (UniqueName: \"kubernetes.io/projected/80ee90cc-3ecf-40be-9d34-511fb0b42721-kube-api-access-fgntb\") pod \"80ee90cc-3ecf-40be-9d34-511fb0b42721\" (UID: \"80ee90cc-3ecf-40be-9d34-511fb0b42721\") "
Nov 03 16:22:03.502589 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.502557    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ee90cc-3ecf-40be-9d34-511fb0b42721-plugins-config-volume" pod "80ee90cc-3ecf-40be-9d34-511fb0b42721" (UID: "80ee90cc-3ecf-40be-9d34-511fb0b42721"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:03.502589 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.502584    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde0ec03-70f8-47bb-af69-6a56524d425f-plugins-config-volume" pod "dde0ec03-70f8-47bb-af69-6a56524d425f" (UID: "dde0ec03-70f8-47bb-af69-6a56524d425f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:03.503260 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.503236    6506 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ee90cc-3ecf-40be-9d34-511fb0b42721-plugins-config-volume\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:03.503351 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.503267    6506 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde0ec03-70f8-47bb-af69-6a56524d425f-plugins-config-volume\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:03.504549 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.504525    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde0ec03-70f8-47bb-af69-6a56524d425f-kube-api-access-gqf59" pod "dde0ec03-70f8-47bb-af69-6a56524d425f" (UID: "dde0ec03-70f8-47bb-af69-6a56524d425f"). InnerVolumeSpecName "kube-api-access-gqf59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:03.504845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.504782    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ee90cc-3ecf-40be-9d34-511fb0b42721-kube-api-access-fgntb" pod "80ee90cc-3ecf-40be-9d34-511fb0b42721" (UID: "80ee90cc-3ecf-40be-9d34-511fb0b42721"). InnerVolumeSpecName "kube-api-access-fgntb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:03.604511 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.604481    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqf59\" (UniqueName: \"kubernetes.io/projected/dde0ec03-70f8-47bb-af69-6a56524d425f-kube-api-access-gqf59\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:03.604511 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.604508    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgntb\" (UniqueName: \"kubernetes.io/projected/80ee90cc-3ecf-40be-9d34-511fb0b42721-kube-api-access-fgntb\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:03.737365 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.737334    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"]
Nov 03 16:22:03.760480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:03.760452    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c09b17-65997658f5-hhv8g"]
Nov 03 16:22:04.025823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:04.025764    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:04.150228 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:04.149787    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ee90cc-3ecf-40be-9d34-511fb0b42721" path="/var/lib/kubelet/pods/80ee90cc-3ecf-40be-9d34-511fb0b42721/volumes"
Nov 03 16:22:04.258625 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:04.258549    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:04.393998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:04.393963    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n" event={"ID":"dde0ec03-70f8-47bb-af69-6a56524d425f","Type":"ContainerDied","Data":"b4f6a130efc0435a0d3dc7fc35b841e1e91f785e5abb5d291f61131e51777254"}
Nov 03 16:22:04.394186 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:04.393980    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"
Nov 03 16:22:04.394186 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:04.394039    6506 scope.go:122] "RemoveContainer" containerID="c2ea69746fae98fd8d8694fa50e08db9670abda8c0717fd6beb5e3588399bb33"
Nov 03 16:22:04.418601 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:04.418575    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"]
Nov 03 16:22:04.425215 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:04.425188    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c09b17-7cff974445-9sc2n"]
Nov 03 16:22:04.456850 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:04.456781    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:04.787928 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:04.787835    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"]
Nov 03 16:22:04.789368 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:04.789324    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4" podUID="ca2bd3e0-8c45-4a8d-9733-0fc00abb7742" containerName="maas-api" containerID="cri-o://9464c131233cd3d78eeaee9ff7c0beeb74c228ada948e87538e0691f3cafa9a1" gracePeriod=30
Nov 03 16:22:04.893331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:04.893303    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:04.899097 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:04.899066    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4" podUID="ca2bd3e0-8c45-4a8d-9733-0fc00abb7742" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.52:8443/health\": dial tcp 10.133.0.52:8443: connect: connection refused"
Nov 03 16:22:04.956676 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:04.956635    6506 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:04.956883 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:04.956698    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg" podUID="c81a5fc9-15c0-491f-bb7a-939e4485f544" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:05.156494 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.156343    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:05.252149 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.251839    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:22:05.258839 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.258784    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"]
Nov 03 16:22:05.259914 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.259886    6506 cpu_manager.go:414] "containerMap: removing container" podUID="52a4e8de-d68f-4726-8699-d452f998cc64" containerName="revoke-keys"
Nov 03 16:22:05.259914 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.259913    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a4e8de-d68f-4726-8699-d452f998cc64" containerName="revoke-keys"
Nov 03 16:22:05.260111 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.259955    6506 cpu_manager.go:414] "containerMap: removing container" podUID="b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc" containerName="revoke-keys"
Nov 03 16:22:05.260111 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.259963    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc" containerName="revoke-keys"
Nov 03 16:22:05.260111 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.259980    6506 cpu_manager.go:414] "containerMap: removing container" podUID="ca2bd3e0-8c45-4a8d-9733-0fc00abb7742" containerName="maas-api"
Nov 03 16:22:05.260111 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.259987    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2bd3e0-8c45-4a8d-9733-0fc00abb7742" containerName="maas-api"
Nov 03 16:22:05.260111 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.260010    6506 cpu_manager.go:414] "containerMap: removing container" podUID="dde0ec03-70f8-47bb-af69-6a56524d425f" containerName="payload-processing"
Nov 03 16:22:05.260111 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.260018    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde0ec03-70f8-47bb-af69-6a56524d425f" containerName="payload-processing"
Nov 03 16:22:05.260111 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.260049    6506 cpu_manager.go:414] "containerMap: removing container" podUID="80ee90cc-3ecf-40be-9d34-511fb0b42721" containerName="payload-pre-processing"
Nov 03 16:22:05.260111 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.260056    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ee90cc-3ecf-40be-9d34-511fb0b42721" containerName="payload-pre-processing"
Nov 03 16:22:05.261937 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.260179    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ee90cc-3ecf-40be-9d34-511fb0b42721" containerName="payload-pre-processing"
Nov 03 16:22:05.261937 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.260194    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a4e8de-d68f-4726-8699-d452f998cc64" containerName="revoke-keys"
Nov 03 16:22:05.261937 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.260208    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc" containerName="revoke-keys"
Nov 03 16:22:05.261937 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.260217    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2bd3e0-8c45-4a8d-9733-0fc00abb7742" containerName="maas-api"
Nov 03 16:22:05.261937 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.260228    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde0ec03-70f8-47bb-af69-6a56524d425f" containerName="payload-processing"
Nov 03 16:22:05.264047 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.264028    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"
Nov 03 16:22:05.269278 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.269253    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:22:05.273150 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.273014    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:22:05.280982 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.280460    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"]
Nov 03 16:22:05.322343 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.322175    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:05.350497 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.350466    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"]
Nov 03 16:22:05.357847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.357822    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c09b17-4jz4m"]
Nov 03 16:22:05.373449 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.373425    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-maas-api-tls\") pod \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") "
Nov 03 16:22:05.373593 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.373485    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-openshift-service-ca\") pod \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") "
Nov 03 16:22:05.373773 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.373749    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-kube-api-access-gmmrz\" (UniqueName: \"kubernetes.io/projected/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-kube-api-access-gmmrz\") pod \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") "
Nov 03 16:22:05.373861 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.373789    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-openshift-service-ca" pod "ca2bd3e0-8c45-4a8d-9733-0fc00abb7742" (UID: "ca2bd3e0-8c45-4a8d-9733-0fc00abb7742"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:05.373861 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.373815    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-metrics-certs\") pod \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\" (UID: \"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742\") "
Nov 03 16:22:05.374235 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.374214    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874bd46d-b2ff-4799-8d5c-6e55b8f79066-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09409a22-qpsp9\" (UID: \"874bd46d-b2ff-4799-8d5c-6e55b8f79066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"
Nov 03 16:22:05.374501 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.374255    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz56n\" (UniqueName: \"kubernetes.io/projected/874bd46d-b2ff-4799-8d5c-6e55b8f79066-kube-api-access-kz56n\") pod \"maas-api-revoke-sub-09409a22-qpsp9\" (UID: \"874bd46d-b2ff-4799-8d5c-6e55b8f79066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"
Nov 03 16:22:05.374501 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.374368    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874bd46d-b2ff-4799-8d5c-6e55b8f79066-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09409a22-qpsp9\" (UID: \"874bd46d-b2ff-4799-8d5c-6e55b8f79066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"
Nov 03 16:22:05.374612 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.374600    6506 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-openshift-service-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:05.375725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.375705    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-maas-api-tls" pod "ca2bd3e0-8c45-4a8d-9733-0fc00abb7742" (UID: "ca2bd3e0-8c45-4a8d-9733-0fc00abb7742"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:05.375922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.375902    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-metrics-certs" pod "ca2bd3e0-8c45-4a8d-9733-0fc00abb7742" (UID: "ca2bd3e0-8c45-4a8d-9733-0fc00abb7742"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:05.376033 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.376018    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-kube-api-access-gmmrz" pod "ca2bd3e0-8c45-4a8d-9733-0fc00abb7742" (UID: "ca2bd3e0-8c45-4a8d-9733-0fc00abb7742"). InnerVolumeSpecName "kube-api-access-gmmrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:05.404406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.404369    6506 generic.go:356] "Generic (PLEG): container finished" podID="ca2bd3e0-8c45-4a8d-9733-0fc00abb7742" containerID="9464c131233cd3d78eeaee9ff7c0beeb74c228ada948e87538e0691f3cafa9a1" exitCode=0
Nov 03 16:22:05.404544 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.404419    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"
Nov 03 16:22:05.404544 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.404450    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4" event={"ID":"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742","Type":"ContainerDied","Data":"9464c131233cd3d78eeaee9ff7c0beeb74c228ada948e87538e0691f3cafa9a1"}
Nov 03 16:22:05.404544 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.404500    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4" event={"ID":"ca2bd3e0-8c45-4a8d-9733-0fc00abb7742","Type":"ContainerDied","Data":"c97b5665e2129e890bd4af850a5a749aa832bb21d22f6de7a79d1ccb112cd2f9"}
Nov 03 16:22:05.404544 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.404529    6506 scope.go:122] "RemoveContainer" containerID="9464c131233cd3d78eeaee9ff7c0beeb74c228ada948e87538e0691f3cafa9a1"
Nov 03 16:22:05.416071 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.416048    6506 scope.go:122] "RemoveContainer" containerID="9464c131233cd3d78eeaee9ff7c0beeb74c228ada948e87538e0691f3cafa9a1"
Nov 03 16:22:05.416348 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:05.416330    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9464c131233cd3d78eeaee9ff7c0beeb74c228ada948e87538e0691f3cafa9a1\": container with ID starting with 9464c131233cd3d78eeaee9ff7c0beeb74c228ada948e87538e0691f3cafa9a1 not found: ID does not exist" containerID="9464c131233cd3d78eeaee9ff7c0beeb74c228ada948e87538e0691f3cafa9a1"
Nov 03 16:22:05.416432 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.416356    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9464c131233cd3d78eeaee9ff7c0beeb74c228ada948e87538e0691f3cafa9a1"} err="failed to get container status \"9464c131233cd3d78eeaee9ff7c0beeb74c228ada948e87538e0691f3cafa9a1\": rpc error: code = NotFound desc = could not find container \"9464c131233cd3d78eeaee9ff7c0beeb74c228ada948e87538e0691f3cafa9a1\": container with ID starting with 9464c131233cd3d78eeaee9ff7c0beeb74c228ada948e87538e0691f3cafa9a1 not found: ID does not exist"
Nov 03 16:22:05.430115 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.430089    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"]
Nov 03 16:22:05.437610 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.437583    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c09b17-5df6595c44-mx5g4"]
Nov 03 16:22:05.475587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.475556    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874bd46d-b2ff-4799-8d5c-6e55b8f79066-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09409a22-qpsp9\" (UID: \"874bd46d-b2ff-4799-8d5c-6e55b8f79066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"
Nov 03 16:22:05.475587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.475593    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz56n\" (UniqueName: \"kubernetes.io/projected/874bd46d-b2ff-4799-8d5c-6e55b8f79066-kube-api-access-kz56n\") pod \"maas-api-revoke-sub-09409a22-qpsp9\" (UID: \"874bd46d-b2ff-4799-8d5c-6e55b8f79066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"
Nov 03 16:22:05.475845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.475692    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874bd46d-b2ff-4799-8d5c-6e55b8f79066-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09409a22-qpsp9\" (UID: \"874bd46d-b2ff-4799-8d5c-6e55b8f79066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"
Nov 03 16:22:05.475845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.475840    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-maas-api-tls\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:05.476017 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.475854    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmmrz\" (UniqueName: \"kubernetes.io/projected/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-kube-api-access-gmmrz\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:05.476017 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.475865    6506 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742-metrics-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:05.476423 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.476403    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874bd46d-b2ff-4799-8d5c-6e55b8f79066-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09409a22-qpsp9\" (UID: \"874bd46d-b2ff-4799-8d5c-6e55b8f79066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"
Nov 03 16:22:05.478181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.478160    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874bd46d-b2ff-4799-8d5c-6e55b8f79066-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09409a22-qpsp9\" (UID: \"874bd46d-b2ff-4799-8d5c-6e55b8f79066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"
Nov 03 16:22:05.499025 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.498956    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz56n\" (UniqueName: \"kubernetes.io/projected/874bd46d-b2ff-4799-8d5c-6e55b8f79066-kube-api-access-kz56n\") pod \"maas-api-revoke-sub-09409a22-qpsp9\" (UID: \"874bd46d-b2ff-4799-8d5c-6e55b8f79066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"
Nov 03 16:22:05.518722 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.518654    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:05.587336 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.587254    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"
Nov 03 16:22:05.841176 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.841055    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:05.870915 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:22:05.870882    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874bd46d_b2ff_4799_8d5c_6e55b8f79066.slice/crio-a648db7b5ac8cdbc01d20a628fc66a8d91109a608220d1dde67da491f5c5d17c WatchSource:0}: Error finding container a648db7b5ac8cdbc01d20a628fc66a8d91109a608220d1dde67da491f5c5d17c: Status 404 returned error can't find the container with id a648db7b5ac8cdbc01d20a628fc66a8d91109a608220d1dde67da491f5c5d17c
Nov 03 16:22:05.871013 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:05.870955    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"]
Nov 03 16:22:06.108664 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:06.108588    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5553aa2c-597b-4bbb-8559-b97e50c48bd4" path="/var/lib/kubelet/pods/5553aa2c-597b-4bbb-8559-b97e50c48bd4/volumes"
Nov 03 16:22:06.109560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:06.109535    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2bd3e0-8c45-4a8d-9733-0fc00abb7742" path="/var/lib/kubelet/pods/ca2bd3e0-8c45-4a8d-9733-0fc00abb7742/volumes"
Nov 03 16:22:06.110347 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:06.110327    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dde0ec03-70f8-47bb-af69-6a56524d425f" path="/var/lib/kubelet/pods/dde0ec03-70f8-47bb-af69-6a56524d425f/volumes"
Nov 03 16:22:06.331787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:06.331758    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:06.417486 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:06.417398    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9" event={"ID":"874bd46d-b2ff-4799-8d5c-6e55b8f79066","Type":"ContainerStarted","Data":"6d2742712fdaf44f345434f2a5f499d9a12352c595a1124ed797df0a198b427d"}
Nov 03 16:22:06.417486 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:06.417458    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9" event={"ID":"874bd46d-b2ff-4799-8d5c-6e55b8f79066","Type":"ContainerStarted","Data":"a648db7b5ac8cdbc01d20a628fc66a8d91109a608220d1dde67da491f5c5d17c"}
Nov 03 16:22:06.441363 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:06.441315    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9" podStartSLOduration=1.44130169 podStartE2EDuration="1.44130169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:06.439754373 +0000 UTC m=+1140.453152518" watchObservedRunningTime="2026-11-03 16:22:06.44130169 +0000 UTC m=+1140.454699836"
Nov 03 16:22:06.536953 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:06.536917    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"]
Nov 03 16:22:06.700379 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:06.700279    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:06.842866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:06.842001    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"]
Nov 03 16:22:06.842866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:06.842273    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2" podUID="02798929-b745-48ba-8db3-31b20be5367b" containerName="istio-proxy" containerID="cri-o://b7e9fe6c0f5024c5c5bfc55f16fd8a664c7056eba751680a581a5bd410bfbc4e" gracePeriod=30
Nov 03 16:22:06.853525 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:06.853496    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"]
Nov 03 16:22:06.853962 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:06.853931    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz" podUID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" containerName="istio-proxy" containerID="cri-o://534bbf6cd27b3d9c1333058425948734bf1a2a66efcf70b283b20fe2b5df370f" gracePeriod=30
Nov 03 16:22:06.870892 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:06.870025    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:07.428757 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.428703    6506 generic.go:356] "Generic (PLEG): container finished" podID="874bd46d-b2ff-4799-8d5c-6e55b8f79066" containerID="6d2742712fdaf44f345434f2a5f499d9a12352c595a1124ed797df0a198b427d" exitCode=0
Nov 03 16:22:07.428974 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.428951    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9" event={"ID":"874bd46d-b2ff-4799-8d5c-6e55b8f79066","Type":"ContainerDied","Data":"6d2742712fdaf44f345434f2a5f499d9a12352c595a1124ed797df0a198b427d"}
Nov 03 16:22:07.431302 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.431275    6506 generic.go:356] "Generic (PLEG): container finished" podID="05788e58-6105-4a9d-812f-5fcfc45a820c" containerID="7e55e78fdc2a481312d16b2c3ff7739adcda4aa78cdc3be83a9dafa4f4835f53" exitCode=0
Nov 03 16:22:07.431419 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.431337    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t" event={"ID":"05788e58-6105-4a9d-812f-5fcfc45a820c","Type":"ContainerDied","Data":"7e55e78fdc2a481312d16b2c3ff7739adcda4aa78cdc3be83a9dafa4f4835f53"}
Nov 03 16:22:07.437830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.434673    6506 generic.go:356] "Generic (PLEG): container finished" podID="c81a5fc9-15c0-491f-bb7a-939e4485f544" containerID="03d73ada7084ffd770703c631df0f207ae30072fab9a450fa4e19f6a156c425d" exitCode=0
Nov 03 16:22:07.437830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.434781    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg" event={"ID":"c81a5fc9-15c0-491f-bb7a-939e4485f544","Type":"ContainerDied","Data":"03d73ada7084ffd770703c631df0f207ae30072fab9a450fa4e19f6a156c425d"}
Nov 03 16:22:07.545181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.545151    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:22:07.592223 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.592190    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:22:07.707927 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.707882    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81a5fc9-15c0-491f-bb7a-939e4485f544-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81a5fc9-15c0-491f-bb7a-939e4485f544-istiod-ca-cert\") pod \"c81a5fc9-15c0-491f-bb7a-939e4485f544\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") "
Nov 03 16:22:07.708119 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.707941    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-token\" (UniqueName: \"kubernetes.io/projected/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-token\") pod \"05788e58-6105-4a9d-812f-5fcfc45a820c\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") "
Nov 03 16:22:07.708119 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708049    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-workload-socket\") pod \"c81a5fc9-15c0-491f-bb7a-939e4485f544\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") "
Nov 03 16:22:07.708119 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708094    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-credential-socket\") pod \"05788e58-6105-4a9d-812f-5fcfc45a820c\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") "
Nov 03 16:22:07.708491 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708130    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-token\" (UniqueName: \"kubernetes.io/projected/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-token\") pod \"c81a5fc9-15c0-491f-bb7a-939e4485f544\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") "
Nov 03 16:22:07.708491 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708184    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-workload-certs\") pod \"c81a5fc9-15c0-491f-bb7a-939e4485f544\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") "
Nov 03 16:22:07.708491 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708313    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81a5fc9-15c0-491f-bb7a-939e4485f544-istiod-ca-cert" pod "c81a5fc9-15c0-491f-bb7a-939e4485f544" (UID: "c81a5fc9-15c0-491f-bb7a-939e4485f544"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:07.708491 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708374    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-data\") pod \"05788e58-6105-4a9d-812f-5fcfc45a820c\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") "
Nov 03 16:22:07.708491 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708400    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81a5fc9-15c0-491f-bb7a-939e4485f544-kube-api-access-vfhjd\" (UniqueName: \"kubernetes.io/projected/c81a5fc9-15c0-491f-bb7a-939e4485f544-kube-api-access-vfhjd\") pod \"c81a5fc9-15c0-491f-bb7a-939e4485f544\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") "
Nov 03 16:22:07.708491 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708469    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05788e58-6105-4a9d-812f-5fcfc45a820c-kube-api-access-nwc7q\" (UniqueName: \"kubernetes.io/projected/05788e58-6105-4a9d-812f-5fcfc45a820c-kube-api-access-nwc7q\") pod \"05788e58-6105-4a9d-812f-5fcfc45a820c\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") "
Nov 03 16:22:07.708823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708507    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05788e58-6105-4a9d-812f-5fcfc45a820c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05788e58-6105-4a9d-812f-5fcfc45a820c-istiod-ca-cert\") pod \"05788e58-6105-4a9d-812f-5fcfc45a820c\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") "
Nov 03 16:22:07.708823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708543    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-envoy\") pod \"05788e58-6105-4a9d-812f-5fcfc45a820c\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") "
Nov 03 16:22:07.708823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708573    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-podinfo\") pod \"c81a5fc9-15c0-491f-bb7a-939e4485f544\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") "
Nov 03 16:22:07.708823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708578    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-credential-socket" pod "05788e58-6105-4a9d-812f-5fcfc45a820c" (UID: "05788e58-6105-4a9d-812f-5fcfc45a820c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:07.708823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708599    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-data\") pod \"c81a5fc9-15c0-491f-bb7a-939e4485f544\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") "
Nov 03 16:22:07.708823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708686    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-workload-socket" pod "c81a5fc9-15c0-491f-bb7a-939e4485f544" (UID: "c81a5fc9-15c0-491f-bb7a-939e4485f544"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:07.708823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708689    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-podinfo\") pod \"05788e58-6105-4a9d-812f-5fcfc45a820c\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") "
Nov 03 16:22:07.708823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708725    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-workload-certs" pod "c81a5fc9-15c0-491f-bb7a-939e4485f544" (UID: "c81a5fc9-15c0-491f-bb7a-939e4485f544"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:07.708823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708782    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-data" pod "c81a5fc9-15c0-491f-bb7a-939e4485f544" (UID: "c81a5fc9-15c0-491f-bb7a-939e4485f544"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:07.708823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708784    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-credential-socket\") pod \"c81a5fc9-15c0-491f-bb7a-939e4485f544\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") "
Nov 03 16:22:07.709370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708859    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-workload-certs\") pod \"05788e58-6105-4a9d-812f-5fcfc45a820c\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") "
Nov 03 16:22:07.709370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708924    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-envoy\") pod \"c81a5fc9-15c0-491f-bb7a-939e4485f544\" (UID: \"c81a5fc9-15c0-491f-bb7a-939e4485f544\") "
Nov 03 16:22:07.709370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708955    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-data" pod "05788e58-6105-4a9d-812f-5fcfc45a820c" (UID: "05788e58-6105-4a9d-812f-5fcfc45a820c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:07.709370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708965    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-credential-socket" pod "c81a5fc9-15c0-491f-bb7a-939e4485f544" (UID: "c81a5fc9-15c0-491f-bb7a-939e4485f544"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:07.709370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.708979    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-workload-socket\") pod \"05788e58-6105-4a9d-812f-5fcfc45a820c\" (UID: \"05788e58-6105-4a9d-812f-5fcfc45a820c\") "
Nov 03 16:22:07.709370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.709182    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-workload-certs" pod "05788e58-6105-4a9d-812f-5fcfc45a820c" (UID: "05788e58-6105-4a9d-812f-5fcfc45a820c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:07.710096 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.710063    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05788e58-6105-4a9d-812f-5fcfc45a820c-istiod-ca-cert" pod "05788e58-6105-4a9d-812f-5fcfc45a820c" (UID: "05788e58-6105-4a9d-812f-5fcfc45a820c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:07.710232 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.710205    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.710232 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.710212    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-workload-socket" pod "05788e58-6105-4a9d-812f-5fcfc45a820c" (UID: "05788e58-6105-4a9d-812f-5fcfc45a820c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:07.710376 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.710226    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.710376 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.710261    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05788e58-6105-4a9d-812f-5fcfc45a820c-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.710376 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.710278    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.710376 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.710293    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.710376 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.710308    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.710376 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.710323    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81a5fc9-15c0-491f-bb7a-939e4485f544-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.710376 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.710338    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.710376 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.710352    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.711405 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.711381    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-envoy" pod "05788e58-6105-4a9d-812f-5fcfc45a820c" (UID: "05788e58-6105-4a9d-812f-5fcfc45a820c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:07.711607 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.711579    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-podinfo" pod "05788e58-6105-4a9d-812f-5fcfc45a820c" (UID: "05788e58-6105-4a9d-812f-5fcfc45a820c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:07.711755 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.711739    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-token" pod "05788e58-6105-4a9d-812f-5fcfc45a820c" (UID: "05788e58-6105-4a9d-812f-5fcfc45a820c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:07.712043 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.712025    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-podinfo" pod "c81a5fc9-15c0-491f-bb7a-939e4485f544" (UID: "c81a5fc9-15c0-491f-bb7a-939e4485f544"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:07.712529 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.712513    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81a5fc9-15c0-491f-bb7a-939e4485f544-kube-api-access-vfhjd" pod "c81a5fc9-15c0-491f-bb7a-939e4485f544" (UID: "c81a5fc9-15c0-491f-bb7a-939e4485f544"). InnerVolumeSpecName "kube-api-access-vfhjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:07.712638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.712615    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-envoy" pod "c81a5fc9-15c0-491f-bb7a-939e4485f544" (UID: "c81a5fc9-15c0-491f-bb7a-939e4485f544"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:07.713001 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.712984    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-token" pod "c81a5fc9-15c0-491f-bb7a-939e4485f544" (UID: "c81a5fc9-15c0-491f-bb7a-939e4485f544"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:07.713059 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.713033    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05788e58-6105-4a9d-812f-5fcfc45a820c-kube-api-access-nwc7q" pod "05788e58-6105-4a9d-812f-5fcfc45a820c" (UID: "05788e58-6105-4a9d-812f-5fcfc45a820c"). InnerVolumeSpecName "kube-api-access-nwc7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:07.827787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.825007    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.827787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.825054    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.827787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.825069    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.827787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.825085    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.827787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.825185    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.827787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.825201    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfhjd\" (UniqueName: \"kubernetes.io/projected/c81a5fc9-15c0-491f-bb7a-939e4485f544-kube-api-access-vfhjd\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.827787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.825217    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwc7q\" (UniqueName: \"kubernetes.io/projected/05788e58-6105-4a9d-812f-5fcfc45a820c-kube-api-access-nwc7q\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.827787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.825231    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05788e58-6105-4a9d-812f-5fcfc45a820c-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.827787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.825245    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81a5fc9-15c0-491f-bb7a-939e4485f544-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.835529 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:07.835502    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:08.443731 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.443654    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"
Nov 03 16:22:08.444192 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.443653    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t" event={"ID":"05788e58-6105-4a9d-812f-5fcfc45a820c","Type":"ContainerDied","Data":"6f9e055149a8c0992107faf5adb5e318ebf27b83fd9fe1f5f883ce5eb4bf3f93"}
Nov 03 16:22:08.444192 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.443826    6506 scope.go:122] "RemoveContainer" containerID="7e55e78fdc2a481312d16b2c3ff7739adcda4aa78cdc3be83a9dafa4f4835f53"
Nov 03 16:22:08.445267 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.445213    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"
Nov 03 16:22:08.445267 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.445232    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg" event={"ID":"c81a5fc9-15c0-491f-bb7a-939e4485f544","Type":"ContainerDied","Data":"b91a9eddc0113c57541dc9ae70154d27279b1cb2532a389d4e8e3dd2fc59f7c8"}
Nov 03 16:22:08.454815 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.454781    6506 scope.go:122] "RemoveContainer" containerID="03d73ada7084ffd770703c631df0f207ae30072fab9a450fa4e19f6a156c425d"
Nov 03 16:22:08.516902 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.516871    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"]
Nov 03 16:22:08.527521 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.527486    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-tw7tg"]
Nov 03 16:22:08.541218 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.541187    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"]
Nov 03 16:22:08.544745 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.544718    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-99v8t"]
Nov 03 16:22:08.602842 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.602814    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"
Nov 03 16:22:08.636009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.635977    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874bd46d-b2ff-4799-8d5c-6e55b8f79066-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874bd46d-b2ff-4799-8d5c-6e55b8f79066-maas-api-openshift-ca\") pod \"874bd46d-b2ff-4799-8d5c-6e55b8f79066\" (UID: \"874bd46d-b2ff-4799-8d5c-6e55b8f79066\") "
Nov 03 16:22:08.636160 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.636130    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874bd46d-b2ff-4799-8d5c-6e55b8f79066-kube-api-access-kz56n\" (UniqueName: \"kubernetes.io/projected/874bd46d-b2ff-4799-8d5c-6e55b8f79066-kube-api-access-kz56n\") pod \"874bd46d-b2ff-4799-8d5c-6e55b8f79066\" (UID: \"874bd46d-b2ff-4799-8d5c-6e55b8f79066\") "
Nov 03 16:22:08.636242 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.636214    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/874bd46d-b2ff-4799-8d5c-6e55b8f79066-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874bd46d-b2ff-4799-8d5c-6e55b8f79066-maas-api-xks-ca\") pod \"874bd46d-b2ff-4799-8d5c-6e55b8f79066\" (UID: \"874bd46d-b2ff-4799-8d5c-6e55b8f79066\") "
Nov 03 16:22:08.636369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.636343    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874bd46d-b2ff-4799-8d5c-6e55b8f79066-maas-api-openshift-ca" pod "874bd46d-b2ff-4799-8d5c-6e55b8f79066" (UID: "874bd46d-b2ff-4799-8d5c-6e55b8f79066"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:08.636814 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.636777    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874bd46d-b2ff-4799-8d5c-6e55b8f79066-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:08.638390 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.638363    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/874bd46d-b2ff-4799-8d5c-6e55b8f79066-maas-api-xks-ca" pod "874bd46d-b2ff-4799-8d5c-6e55b8f79066" (UID: "874bd46d-b2ff-4799-8d5c-6e55b8f79066"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:08.638860 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.638843    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874bd46d-b2ff-4799-8d5c-6e55b8f79066-kube-api-access-kz56n" pod "874bd46d-b2ff-4799-8d5c-6e55b8f79066" (UID: "874bd46d-b2ff-4799-8d5c-6e55b8f79066"). InnerVolumeSpecName "kube-api-access-kz56n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:08.738606 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.738527    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz56n\" (UniqueName: \"kubernetes.io/projected/874bd46d-b2ff-4799-8d5c-6e55b8f79066-kube-api-access-kz56n\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:08.738606 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.738559    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874bd46d-b2ff-4799-8d5c-6e55b8f79066-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:08.805187 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:08.805149    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:09.311662 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:09.311631    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:09.453744 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:09.453711    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"
Nov 03 16:22:09.454220 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:09.453758    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9" event={"ID":"874bd46d-b2ff-4799-8d5c-6e55b8f79066","Type":"ContainerDied","Data":"a648db7b5ac8cdbc01d20a628fc66a8d91109a608220d1dde67da491f5c5d17c"}
Nov 03 16:22:09.454220 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:09.453826    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a648db7b5ac8cdbc01d20a628fc66a8d91109a608220d1dde67da491f5c5d17c"
Nov 03 16:22:09.466158 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:09.466136    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:09.700369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:09.700285    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:09.907513 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:09.907441    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:10.105949 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:10.105914    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05788e58-6105-4a9d-812f-5fcfc45a820c" path="/var/lib/kubelet/pods/05788e58-6105-4a9d-812f-5fcfc45a820c/volumes"
Nov 03 16:22:10.106616 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:10.106590    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81a5fc9-15c0-491f-bb7a-939e4485f544" path="/var/lib/kubelet/pods/c81a5fc9-15c0-491f-bb7a-939e4485f544/volumes"
Nov 03 16:22:10.899993 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:10.899959    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:10.900390 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:10.900021    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz" podUID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:11.067386 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:11.067349    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:11.875535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:11.875502    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c" podUID="381eb05d-c83e-49b3-89b3-935ea62fb4e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:22:11.923925 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:11.923889    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:12.160934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.160909    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:22:12.186585 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.185976    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-workload-socket\") pod \"02798929-b745-48ba-8db3-31b20be5367b\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") "
Nov 03 16:22:12.186585 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.186090    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02798929-b745-48ba-8db3-31b20be5367b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02798929-b745-48ba-8db3-31b20be5367b-istiod-ca-cert\") pod \"02798929-b745-48ba-8db3-31b20be5367b\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") "
Nov 03 16:22:12.186585 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.186125    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-istio-data\") pod \"02798929-b745-48ba-8db3-31b20be5367b\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") "
Nov 03 16:22:12.186585 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.186178    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-credential-socket\") pod \"02798929-b745-48ba-8db3-31b20be5367b\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") "
Nov 03 16:22:12.186585 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.186262    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02798929-b745-48ba-8db3-31b20be5367b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02798929-b745-48ba-8db3-31b20be5367b-istio-podinfo\") pod \"02798929-b745-48ba-8db3-31b20be5367b\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") "
Nov 03 16:22:12.186585 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.186309    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-istio-envoy\") pod \"02798929-b745-48ba-8db3-31b20be5367b\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") "
Nov 03 16:22:12.186585 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.186341    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02798929-b745-48ba-8db3-31b20be5367b-istio-token\" (UniqueName: \"kubernetes.io/projected/02798929-b745-48ba-8db3-31b20be5367b-istio-token\") pod \"02798929-b745-48ba-8db3-31b20be5367b\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") "
Nov 03 16:22:12.187019 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.186896    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-workload-certs" pod "02798929-b745-48ba-8db3-31b20be5367b" (UID: "02798929-b745-48ba-8db3-31b20be5367b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:12.187141 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.187120    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-workload-socket" pod "02798929-b745-48ba-8db3-31b20be5367b" (UID: "02798929-b745-48ba-8db3-31b20be5367b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:12.187365 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.187347    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02798929-b745-48ba-8db3-31b20be5367b-istiod-ca-cert" pod "02798929-b745-48ba-8db3-31b20be5367b" (UID: "02798929-b745-48ba-8db3-31b20be5367b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:12.187554 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.187537    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-istio-data" pod "02798929-b745-48ba-8db3-31b20be5367b" (UID: "02798929-b745-48ba-8db3-31b20be5367b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:12.187858 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.187686    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-credential-socket" pod "02798929-b745-48ba-8db3-31b20be5367b" (UID: "02798929-b745-48ba-8db3-31b20be5367b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:12.188439 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.188414    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"]
Nov 03 16:22:12.188688 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.186369    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-workload-certs\") pod \"02798929-b745-48ba-8db3-31b20be5367b\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") "
Nov 03 16:22:12.188881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.188779    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02798929-b745-48ba-8db3-31b20be5367b-kube-api-access-vb9wg\" (UniqueName: \"kubernetes.io/projected/02798929-b745-48ba-8db3-31b20be5367b-kube-api-access-vb9wg\") pod \"02798929-b745-48ba-8db3-31b20be5367b\" (UID: \"02798929-b745-48ba-8db3-31b20be5367b\") "
Nov 03 16:22:12.189312 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.189293    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.189394 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.189316    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02798929-b745-48ba-8db3-31b20be5367b-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.189394 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.189330    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.189394 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.189343    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.189394 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.189356    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.206489 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.206437    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02798929-b745-48ba-8db3-31b20be5367b-kube-api-access-vb9wg" pod "02798929-b745-48ba-8db3-31b20be5367b" (UID: "02798929-b745-48ba-8db3-31b20be5367b"). InnerVolumeSpecName "kube-api-access-vb9wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:12.206777 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.206741    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02798929-b745-48ba-8db3-31b20be5367b-istio-podinfo" pod "02798929-b745-48ba-8db3-31b20be5367b" (UID: "02798929-b745-48ba-8db3-31b20be5367b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:12.206947 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.206921    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-istio-envoy" pod "02798929-b745-48ba-8db3-31b20be5367b" (UID: "02798929-b745-48ba-8db3-31b20be5367b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:12.212141 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.210931    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02798929-b745-48ba-8db3-31b20be5367b-istio-token" pod "02798929-b745-48ba-8db3-31b20be5367b" (UID: "02798929-b745-48ba-8db3-31b20be5367b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:12.220337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.220311    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:22:12.228819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.227681    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:12.290544 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.290507    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istiod-ca-cert\") pod \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") "
Nov 03 16:22:12.290737 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.290560    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-data\") pod \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") "
Nov 03 16:22:12.290737 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.290630    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-token\" (UniqueName: \"kubernetes.io/projected/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-token\") pod \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") "
Nov 03 16:22:12.290737 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.290659    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-credential-socket\") pod \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") "
Nov 03 16:22:12.290737 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.290693    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-podinfo\") pod \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") "
Nov 03 16:22:12.290998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.290851    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-workload-socket\") pod \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") "
Nov 03 16:22:12.290998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.290878    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-envoy\") pod \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") "
Nov 03 16:22:12.290998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.290938    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52baf237-4a39-4822-b1b4-4d9a6bd36b85-kube-api-access-96t87\" (UniqueName: \"kubernetes.io/projected/52baf237-4a39-4822-b1b4-4d9a6bd36b85-kube-api-access-96t87\") pod \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") "
Nov 03 16:22:12.291161 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.291001    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-workload-certs\") pod \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\" (UID: \"52baf237-4a39-4822-b1b4-4d9a6bd36b85\") "
Nov 03 16:22:12.291479 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.291454    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02798929-b745-48ba-8db3-31b20be5367b-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.291614 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.291483    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02798929-b745-48ba-8db3-31b20be5367b-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.291614 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.291497    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02798929-b745-48ba-8db3-31b20be5367b-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.291614 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.291511    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb9wg\" (UniqueName: \"kubernetes.io/projected/02798929-b745-48ba-8db3-31b20be5367b-kube-api-access-vb9wg\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.291776 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.291616    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-data" pod "52baf237-4a39-4822-b1b4-4d9a6bd36b85" (UID: "52baf237-4a39-4822-b1b4-4d9a6bd36b85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:12.291776 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.291717    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-workload-certs" pod "52baf237-4a39-4822-b1b4-4d9a6bd36b85" (UID: "52baf237-4a39-4822-b1b4-4d9a6bd36b85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:12.292225 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.291995    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istiod-ca-cert" pod "52baf237-4a39-4822-b1b4-4d9a6bd36b85" (UID: "52baf237-4a39-4822-b1b4-4d9a6bd36b85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:12.292545 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.292322    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-workload-socket" pod "52baf237-4a39-4822-b1b4-4d9a6bd36b85" (UID: "52baf237-4a39-4822-b1b4-4d9a6bd36b85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:12.292787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.292768    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-credential-socket" pod "52baf237-4a39-4822-b1b4-4d9a6bd36b85" (UID: "52baf237-4a39-4822-b1b4-4d9a6bd36b85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:12.294812 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.294479    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52baf237-4a39-4822-b1b4-4d9a6bd36b85-kube-api-access-96t87" pod "52baf237-4a39-4822-b1b4-4d9a6bd36b85" (UID: "52baf237-4a39-4822-b1b4-4d9a6bd36b85"). InnerVolumeSpecName "kube-api-access-96t87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:12.298185 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.298142    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-token" pod "52baf237-4a39-4822-b1b4-4d9a6bd36b85" (UID: "52baf237-4a39-4822-b1b4-4d9a6bd36b85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:12.298299 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.298278    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-envoy" pod "52baf237-4a39-4822-b1b4-4d9a6bd36b85" (UID: "52baf237-4a39-4822-b1b4-4d9a6bd36b85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:12.299377 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.299351    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-podinfo" pod "52baf237-4a39-4822-b1b4-4d9a6bd36b85" (UID: "52baf237-4a39-4822-b1b4-4d9a6bd36b85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:12.392694 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.392623    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.392694 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.392650    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.392694 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.392661    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96t87\" (UniqueName: \"kubernetes.io/projected/52baf237-4a39-4822-b1b4-4d9a6bd36b85-kube-api-access-96t87\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.392694 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.392669    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.392694 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.392678    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.392694 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.392686    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.392694 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.392694    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.393041 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.392702    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52baf237-4a39-4822-b1b4-4d9a6bd36b85-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.393041 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.392710    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52baf237-4a39-4822-b1b4-4d9a6bd36b85-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.420966 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.420931    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:12.479195 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.479164    6506 generic.go:356] "Generic (PLEG): container finished" podID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" containerID="534bbf6cd27b3d9c1333058425948734bf1a2a66efcf70b283b20fe2b5df370f" exitCode=0
Nov 03 16:22:12.479382 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.479218    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"
Nov 03 16:22:12.479382 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.479263    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz" event={"ID":"52baf237-4a39-4822-b1b4-4d9a6bd36b85","Type":"ContainerDied","Data":"534bbf6cd27b3d9c1333058425948734bf1a2a66efcf70b283b20fe2b5df370f"}
Nov 03 16:22:12.479382 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.479317    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz" event={"ID":"52baf237-4a39-4822-b1b4-4d9a6bd36b85","Type":"ContainerDied","Data":"d21e7437e2c673fba2261de6b3abbea00a2ffbe08b64bbdb471d4f3b85db3f88"}
Nov 03 16:22:12.479382 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.479343    6506 scope.go:122] "RemoveContainer" containerID="534bbf6cd27b3d9c1333058425948734bf1a2a66efcf70b283b20fe2b5df370f"
Nov 03 16:22:12.480990 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.480960    6506 generic.go:356] "Generic (PLEG): container finished" podID="02798929-b745-48ba-8db3-31b20be5367b" containerID="b7e9fe6c0f5024c5c5bfc55f16fd8a664c7056eba751680a581a5bd410bfbc4e" exitCode=0
Nov 03 16:22:12.481105 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.481009    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"
Nov 03 16:22:12.481166 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.481144    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2" event={"ID":"02798929-b745-48ba-8db3-31b20be5367b","Type":"ContainerDied","Data":"b7e9fe6c0f5024c5c5bfc55f16fd8a664c7056eba751680a581a5bd410bfbc4e"}
Nov 03 16:22:12.481205 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.481178    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2" event={"ID":"02798929-b745-48ba-8db3-31b20be5367b","Type":"ContainerDied","Data":"994f4332780eeef019151cbd9d4fcd2daaa3e99bbc733159d19425523ba52d86"}
Nov 03 16:22:12.492016 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.491993    6506 scope.go:122] "RemoveContainer" containerID="58e19e8ed66f23c5896cde17db008e9264921b1577f320b0d3ca6a554744c5e9"
Nov 03 16:22:12.501888 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.501873    6506 scope.go:122] "RemoveContainer" containerID="534bbf6cd27b3d9c1333058425948734bf1a2a66efcf70b283b20fe2b5df370f"
Nov 03 16:22:12.502164 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:12.502147    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534bbf6cd27b3d9c1333058425948734bf1a2a66efcf70b283b20fe2b5df370f\": container with ID starting with 534bbf6cd27b3d9c1333058425948734bf1a2a66efcf70b283b20fe2b5df370f not found: ID does not exist" containerID="534bbf6cd27b3d9c1333058425948734bf1a2a66efcf70b283b20fe2b5df370f"
Nov 03 16:22:12.502223 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.502171    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534bbf6cd27b3d9c1333058425948734bf1a2a66efcf70b283b20fe2b5df370f"} err="failed to get container status \"534bbf6cd27b3d9c1333058425948734bf1a2a66efcf70b283b20fe2b5df370f\": rpc error: code = NotFound desc = could not find container \"534bbf6cd27b3d9c1333058425948734bf1a2a66efcf70b283b20fe2b5df370f\": container with ID starting with 534bbf6cd27b3d9c1333058425948734bf1a2a66efcf70b283b20fe2b5df370f not found: ID does not exist"
Nov 03 16:22:12.502223 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.502188    6506 scope.go:122] "RemoveContainer" containerID="58e19e8ed66f23c5896cde17db008e9264921b1577f320b0d3ca6a554744c5e9"
Nov 03 16:22:12.502451 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:12.502434    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e19e8ed66f23c5896cde17db008e9264921b1577f320b0d3ca6a554744c5e9\": container with ID starting with 58e19e8ed66f23c5896cde17db008e9264921b1577f320b0d3ca6a554744c5e9 not found: ID does not exist" containerID="58e19e8ed66f23c5896cde17db008e9264921b1577f320b0d3ca6a554744c5e9"
Nov 03 16:22:12.502487 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.502457    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e19e8ed66f23c5896cde17db008e9264921b1577f320b0d3ca6a554744c5e9"} err="failed to get container status \"58e19e8ed66f23c5896cde17db008e9264921b1577f320b0d3ca6a554744c5e9\": rpc error: code = NotFound desc = could not find container \"58e19e8ed66f23c5896cde17db008e9264921b1577f320b0d3ca6a554744c5e9\": container with ID starting with 58e19e8ed66f23c5896cde17db008e9264921b1577f320b0d3ca6a554744c5e9 not found: ID does not exist"
Nov 03 16:22:12.502487 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.502473    6506 scope.go:122] "RemoveContainer" containerID="b7e9fe6c0f5024c5c5bfc55f16fd8a664c7056eba751680a581a5bd410bfbc4e"
Nov 03 16:22:12.512845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.512827    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"]
Nov 03 16:22:12.516354 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.516329    6506 scope.go:122] "RemoveContainer" containerID="b7e9fe6c0f5024c5c5bfc55f16fd8a664c7056eba751680a581a5bd410bfbc4e"
Nov 03 16:22:12.516732 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:12.516701    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e9fe6c0f5024c5c5bfc55f16fd8a664c7056eba751680a581a5bd410bfbc4e\": container with ID starting with b7e9fe6c0f5024c5c5bfc55f16fd8a664c7056eba751680a581a5bd410bfbc4e not found: ID does not exist" containerID="b7e9fe6c0f5024c5c5bfc55f16fd8a664c7056eba751680a581a5bd410bfbc4e"
Nov 03 16:22:12.516872 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.516738    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e9fe6c0f5024c5c5bfc55f16fd8a664c7056eba751680a581a5bd410bfbc4e"} err="failed to get container status \"b7e9fe6c0f5024c5c5bfc55f16fd8a664c7056eba751680a581a5bd410bfbc4e\": rpc error: code = NotFound desc = could not find container \"b7e9fe6c0f5024c5c5bfc55f16fd8a664c7056eba751680a581a5bd410bfbc4e\": container with ID starting with b7e9fe6c0f5024c5c5bfc55f16fd8a664c7056eba751680a581a5bd410bfbc4e not found: ID does not exist"
Nov 03 16:22:12.534730 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.519685    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sdjqz"]
Nov 03 16:22:12.600820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.595839    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"]
Nov 03 16:22:12.627854 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:12.626833    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8hmf2"]
Nov 03 16:22:13.034229 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.034200    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"
Nov 03 16:22:13.095900 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.095811    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:13.103414 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.103379    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381eb05d-c83e-49b3-89b3-935ea62fb4e1-kube-api-access-rgznh\" (UniqueName: \"kubernetes.io/projected/381eb05d-c83e-49b3-89b3-935ea62fb4e1-kube-api-access-rgznh\") pod \"381eb05d-c83e-49b3-89b3-935ea62fb4e1\" (UID: \"381eb05d-c83e-49b3-89b3-935ea62fb4e1\") "
Nov 03 16:22:13.103587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.103541    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/381eb05d-c83e-49b3-89b3-935ea62fb4e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/381eb05d-c83e-49b3-89b3-935ea62fb4e1-plugins-config-volume\") pod \"381eb05d-c83e-49b3-89b3-935ea62fb4e1\" (UID: \"381eb05d-c83e-49b3-89b3-935ea62fb4e1\") "
Nov 03 16:22:13.104100 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.104054    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381eb05d-c83e-49b3-89b3-935ea62fb4e1-plugins-config-volume" pod "381eb05d-c83e-49b3-89b3-935ea62fb4e1" (UID: "381eb05d-c83e-49b3-89b3-935ea62fb4e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:13.107072 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.107033    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381eb05d-c83e-49b3-89b3-935ea62fb4e1-kube-api-access-rgznh" pod "381eb05d-c83e-49b3-89b3-935ea62fb4e1" (UID: "381eb05d-c83e-49b3-89b3-935ea62fb4e1"). InnerVolumeSpecName "kube-api-access-rgznh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:13.114787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.114764    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"]
Nov 03 16:22:13.115438 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115416    6506 cpu_manager.go:414] "containerMap: removing container" podUID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" containerName="istio-proxy"
Nov 03 16:22:13.115438 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115436    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" containerName="istio-proxy"
Nov 03 16:22:13.115619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115454    6506 cpu_manager.go:414] "containerMap: removing container" podUID="02798929-b745-48ba-8db3-31b20be5367b" containerName="istio-proxy"
Nov 03 16:22:13.115619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115459    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02798929-b745-48ba-8db3-31b20be5367b" containerName="istio-proxy"
Nov 03 16:22:13.115619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115469    6506 cpu_manager.go:414] "containerMap: removing container" podUID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" containerName="istio-proxy"
Nov 03 16:22:13.115619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115474    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" containerName="istio-proxy"
Nov 03 16:22:13.115619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115507    6506 cpu_manager.go:414] "containerMap: removing container" podUID="c81a5fc9-15c0-491f-bb7a-939e4485f544" containerName="istio-proxy"
Nov 03 16:22:13.115619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115515    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81a5fc9-15c0-491f-bb7a-939e4485f544" containerName="istio-proxy"
Nov 03 16:22:13.115619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115524    6506 cpu_manager.go:414] "containerMap: removing container" podUID="381eb05d-c83e-49b3-89b3-935ea62fb4e1" containerName="payload-processing"
Nov 03 16:22:13.115619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115528    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381eb05d-c83e-49b3-89b3-935ea62fb4e1" containerName="payload-processing"
Nov 03 16:22:13.115619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115536    6506 cpu_manager.go:414] "containerMap: removing container" podUID="05788e58-6105-4a9d-812f-5fcfc45a820c" containerName="istio-proxy"
Nov 03 16:22:13.115619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115539    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05788e58-6105-4a9d-812f-5fcfc45a820c" containerName="istio-proxy"
Nov 03 16:22:13.115619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115547    6506 cpu_manager.go:414] "containerMap: removing container" podUID="874bd46d-b2ff-4799-8d5c-6e55b8f79066" containerName="revoke-keys"
Nov 03 16:22:13.115619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115552    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874bd46d-b2ff-4799-8d5c-6e55b8f79066" containerName="revoke-keys"
Nov 03 16:22:13.116291 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115641    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" containerName="istio-proxy"
Nov 03 16:22:13.116291 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115651    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="02798929-b745-48ba-8db3-31b20be5367b" containerName="istio-proxy"
Nov 03 16:22:13.116291 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115658    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" containerName="istio-proxy"
Nov 03 16:22:13.116291 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115669    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81a5fc9-15c0-491f-bb7a-939e4485f544" containerName="istio-proxy"
Nov 03 16:22:13.116291 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115676    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="381eb05d-c83e-49b3-89b3-935ea62fb4e1" containerName="payload-processing"
Nov 03 16:22:13.116291 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115681    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="05788e58-6105-4a9d-812f-5fcfc45a820c" containerName="istio-proxy"
Nov 03 16:22:13.116291 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.115687    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="874bd46d-b2ff-4799-8d5c-6e55b8f79066" containerName="revoke-keys"
Nov 03 16:22:13.119612 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.119593    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"
Nov 03 16:22:13.127371 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.124083    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:22:13.127654 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.127633    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:22:13.140568 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.140542    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"]
Nov 03 16:22:13.204752 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.204722    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qv2\" (UniqueName: \"kubernetes.io/projected/288ab83a-1d57-4f44-800f-a9850aed7772-kube-api-access-z5qv2\") pod \"maas-api-revoke-sub-5f01d0ba-rpgtj\" (UID: \"288ab83a-1d57-4f44-800f-a9850aed7772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"
Nov 03 16:22:13.204918 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.204787    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288ab83a-1d57-4f44-800f-a9850aed7772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f01d0ba-rpgtj\" (UID: \"288ab83a-1d57-4f44-800f-a9850aed7772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"
Nov 03 16:22:13.204978 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.204962    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288ab83a-1d57-4f44-800f-a9850aed7772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f01d0ba-rpgtj\" (UID: \"288ab83a-1d57-4f44-800f-a9850aed7772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"
Nov 03 16:22:13.205027 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.205019    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgznh\" (UniqueName: \"kubernetes.io/projected/381eb05d-c83e-49b3-89b3-935ea62fb4e1-kube-api-access-rgznh\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:13.205071 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.205051    6506 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/381eb05d-c83e-49b3-89b3-935ea62fb4e1-plugins-config-volume\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:13.306639 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.306603    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qv2\" (UniqueName: \"kubernetes.io/projected/288ab83a-1d57-4f44-800f-a9850aed7772-kube-api-access-z5qv2\") pod \"maas-api-revoke-sub-5f01d0ba-rpgtj\" (UID: \"288ab83a-1d57-4f44-800f-a9850aed7772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"
Nov 03 16:22:13.306889 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.306693    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288ab83a-1d57-4f44-800f-a9850aed7772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f01d0ba-rpgtj\" (UID: \"288ab83a-1d57-4f44-800f-a9850aed7772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"
Nov 03 16:22:13.306983 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.306884    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288ab83a-1d57-4f44-800f-a9850aed7772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f01d0ba-rpgtj\" (UID: \"288ab83a-1d57-4f44-800f-a9850aed7772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"
Nov 03 16:22:13.307527 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.307500    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288ab83a-1d57-4f44-800f-a9850aed7772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f01d0ba-rpgtj\" (UID: \"288ab83a-1d57-4f44-800f-a9850aed7772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"
Nov 03 16:22:13.309523 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.309494    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288ab83a-1d57-4f44-800f-a9850aed7772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f01d0ba-rpgtj\" (UID: \"288ab83a-1d57-4f44-800f-a9850aed7772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"
Nov 03 16:22:13.318730 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.318709    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qv2\" (UniqueName: \"kubernetes.io/projected/288ab83a-1d57-4f44-800f-a9850aed7772-kube-api-access-z5qv2\") pod \"maas-api-revoke-sub-5f01d0ba-rpgtj\" (UID: \"288ab83a-1d57-4f44-800f-a9850aed7772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"
Nov 03 16:22:13.419875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.419613    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:13.459745 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.459709    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"
Nov 03 16:22:13.500436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.500400    6506 generic.go:356] "Generic (PLEG): container finished" podID="381eb05d-c83e-49b3-89b3-935ea62fb4e1" containerID="c86b1a39aff6e45af98ab2633d29df483eec7f5621650ed9df88268e0f8d7efd" exitCode=0
Nov 03 16:22:13.500436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.500439    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"
Nov 03 16:22:13.500436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.500446    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c" event={"ID":"381eb05d-c83e-49b3-89b3-935ea62fb4e1","Type":"ContainerDied","Data":"c86b1a39aff6e45af98ab2633d29df483eec7f5621650ed9df88268e0f8d7efd"}
Nov 03 16:22:13.500677 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.500498    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c" event={"ID":"381eb05d-c83e-49b3-89b3-935ea62fb4e1","Type":"ContainerDied","Data":"9f754514801aaf46eea452fcaa0fffd08c56f25ca371734a4aa6678dfda11816"}
Nov 03 16:22:13.500677 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.500528    6506 scope.go:122] "RemoveContainer" containerID="c86b1a39aff6e45af98ab2633d29df483eec7f5621650ed9df88268e0f8d7efd"
Nov 03 16:22:13.515847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.515824    6506 scope.go:122] "RemoveContainer" containerID="c86b1a39aff6e45af98ab2633d29df483eec7f5621650ed9df88268e0f8d7efd"
Nov 03 16:22:13.516200 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:13.516175    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86b1a39aff6e45af98ab2633d29df483eec7f5621650ed9df88268e0f8d7efd\": container with ID starting with c86b1a39aff6e45af98ab2633d29df483eec7f5621650ed9df88268e0f8d7efd not found: ID does not exist" containerID="c86b1a39aff6e45af98ab2633d29df483eec7f5621650ed9df88268e0f8d7efd"
Nov 03 16:22:13.516278 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.516214    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86b1a39aff6e45af98ab2633d29df483eec7f5621650ed9df88268e0f8d7efd"} err="failed to get container status \"c86b1a39aff6e45af98ab2633d29df483eec7f5621650ed9df88268e0f8d7efd\": rpc error: code = NotFound desc = could not find container \"c86b1a39aff6e45af98ab2633d29df483eec7f5621650ed9df88268e0f8d7efd\": container with ID starting with c86b1a39aff6e45af98ab2633d29df483eec7f5621650ed9df88268e0f8d7efd not found: ID does not exist"
Nov 03 16:22:13.531931 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.531903    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"]
Nov 03 16:22:13.538961 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.538907    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-69cedd-564f47bc6c-mbx4c"]
Nov 03 16:22:13.772077 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:13.772049    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"]
Nov 03 16:22:13.776102 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:22:13.776076    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288ab83a_1d57_4f44_800f_a9850aed7772.slice/crio-b1a9963bce4e68818cd63f89c9d441248e2c7d0279f9d62904a0218595156a0f WatchSource:0}: Error finding container b1a9963bce4e68818cd63f89c9d441248e2c7d0279f9d62904a0218595156a0f: Status 404 returned error can't find the container with id b1a9963bce4e68818cd63f89c9d441248e2c7d0279f9d62904a0218595156a0f
Nov 03 16:22:14.105448 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.105414    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02798929-b745-48ba-8db3-31b20be5367b" path="/var/lib/kubelet/pods/02798929-b745-48ba-8db3-31b20be5367b/volumes"
Nov 03 16:22:14.106044 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.106024    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381eb05d-c83e-49b3-89b3-935ea62fb4e1" path="/var/lib/kubelet/pods/381eb05d-c83e-49b3-89b3-935ea62fb4e1/volumes"
Nov 03 16:22:14.106496 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.106476    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52baf237-4a39-4822-b1b4-4d9a6bd36b85" path="/var/lib/kubelet/pods/52baf237-4a39-4822-b1b4-4d9a6bd36b85/volumes"
Nov 03 16:22:14.280356 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.280316    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:14.481740 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.481635    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:14.511972 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.511934    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj" event={"ID":"288ab83a-1d57-4f44-800f-a9850aed7772","Type":"ContainerStarted","Data":"8603f3f07726846ff08ccfa82f369ad3277292a8fc5d0e98f4cb3e32dabf0a6c"}
Nov 03 16:22:14.512172 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.511998    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj" event={"ID":"288ab83a-1d57-4f44-800f-a9850aed7772","Type":"ContainerStarted","Data":"b1a9963bce4e68818cd63f89c9d441248e2c7d0279f9d62904a0218595156a0f"}
Nov 03 16:22:14.549646 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.549569    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj" podStartSLOduration=1.549548113 podStartE2EDuration="1.549548113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:14.533921547 +0000 UTC m=+1148.547319699" watchObservedRunningTime="2026-11-03 16:22:14.549548113 +0000 UTC m=+1148.562946260"
Nov 03 16:22:14.703209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.703173    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"]
Nov 03 16:22:14.706629 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.706599    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-69cedd-msnvj"]
Nov 03 16:22:14.926813 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.926765    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"]
Nov 03 16:22:14.927113 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.927048    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q" podUID="46df749f-6606-477d-85db-cfe8e35bd740" containerName="istio-proxy" containerID="cri-o://2f514dd7a56e21bd532836d14728f4f63d0fc089fe38336ceb6fa1abb996ff42" gracePeriod=30
Nov 03 16:22:14.938710 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.938681    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"]
Nov 03 16:22:14.938919 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.938899    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w" podUID="39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" containerName="istio-proxy" containerID="cri-o://bae9d4a98f92a04a924f676f8f05dc81b0a314e180b77203f67cd2ed32dc6bbc" gracePeriod=30
Nov 03 16:22:14.962056 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.962026    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"]
Nov 03 16:22:14.962233 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:14.962215    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5" podUID="e0ed082c-356f-45b5-80e5-c70a6f5d055d" containerName="istio-proxy" containerID="cri-o://f2136fcf41133c45299562b0c9bf9e4b86d61c40cb9f52092b95fd0a5ac329ac" gracePeriod=30
Nov 03 16:22:15.523775 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:15.523742    6506 generic.go:356] "Generic (PLEG): container finished" podID="288ab83a-1d57-4f44-800f-a9850aed7772" containerID="8603f3f07726846ff08ccfa82f369ad3277292a8fc5d0e98f4cb3e32dabf0a6c" exitCode=0
Nov 03 16:22:15.524277 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:15.523811    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj" event={"ID":"288ab83a-1d57-4f44-800f-a9850aed7772","Type":"ContainerDied","Data":"8603f3f07726846ff08ccfa82f369ad3277292a8fc5d0e98f4cb3e32dabf0a6c"}
Nov 03 16:22:16.142821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:16.139202    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4deb4ddf-30da-4eb3-b56a-237274761339" path="/var/lib/kubelet/pods/4deb4ddf-30da-4eb3-b56a-237274761339/volumes"
Nov 03 16:22:16.330492 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:16.328526    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"]
Nov 03 16:22:16.331288 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:16.331255    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw" podUID="bab2854b-0c08-4417-a5cf-bf67c191b51f" containerName="maas-api" containerID="cri-o://ccb5141d3367ac6dd7970f0f7be77e53af74b3dcc9416bc82779debfc54e6c6c" gracePeriod=30
Nov 03 16:22:16.620618 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:16.592081    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:16.709945 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:16.593877    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 03 16:22:16.709945 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:16.687169    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w" podUID="39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:16.899885 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:16.899033    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:17.251490 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:17.210552    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz" podUID="39813549-9eb7-4cb7-9f08-4bf983b4cebc" containerName="payload-processing" containerID="cri-o://119250b5231ab824d8f19f7131df5e0ceb47586ebb4089e30acd9c790c9f6a29" gracePeriod=25
Nov 03 16:22:17.375963 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:17.375876    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:22:17.375963 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:17.375937    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc" podUID="27399426-17f7-4ea7-aba6-12338b82c6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:22:17.557855 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:17.555046    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:22:17.581837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:17.577496    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:22:17.723839 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:17.694899    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:22:17.752837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:17.744404    6506 generic.go:356] "Generic (PLEG): container finished" podID="bab2854b-0c08-4417-a5cf-bf67c191b51f" containerID="ccb5141d3367ac6dd7970f0f7be77e53af74b3dcc9416bc82779debfc54e6c6c" exitCode=0
Nov 03 16:22:17.752837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:17.744666    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw" event={"ID":"bab2854b-0c08-4417-a5cf-bf67c191b51f","Type":"ContainerDied","Data":"ccb5141d3367ac6dd7970f0f7be77e53af74b3dcc9416bc82779debfc54e6c6c"}
Nov 03 16:22:17.946877 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:17.942720    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:17.983817 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:17.971397    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"
Nov 03 16:22:18.128077 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:18.126960    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288ab83a-1d57-4f44-800f-a9850aed7772-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288ab83a-1d57-4f44-800f-a9850aed7772-maas-api-openshift-ca\") pod \"288ab83a-1d57-4f44-800f-a9850aed7772\" (UID: \"288ab83a-1d57-4f44-800f-a9850aed7772\") "
Nov 03 16:22:18.128077 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:18.127079    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288ab83a-1d57-4f44-800f-a9850aed7772-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288ab83a-1d57-4f44-800f-a9850aed7772-maas-api-xks-ca\") pod \"288ab83a-1d57-4f44-800f-a9850aed7772\" (UID: \"288ab83a-1d57-4f44-800f-a9850aed7772\") "
Nov 03 16:22:18.128077 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:18.127166    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288ab83a-1d57-4f44-800f-a9850aed7772-kube-api-access-z5qv2\" (UniqueName: \"kubernetes.io/projected/288ab83a-1d57-4f44-800f-a9850aed7772-kube-api-access-z5qv2\") pod \"288ab83a-1d57-4f44-800f-a9850aed7772\" (UID: \"288ab83a-1d57-4f44-800f-a9850aed7772\") "
Nov 03 16:22:18.319846 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:18.310299    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288ab83a-1d57-4f44-800f-a9850aed7772-maas-api-openshift-ca" pod "288ab83a-1d57-4f44-800f-a9850aed7772" (UID: "288ab83a-1d57-4f44-800f-a9850aed7772"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:18.506517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:18.460705    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288ab83a-1d57-4f44-800f-a9850aed7772-kube-api-access-z5qv2" pod "288ab83a-1d57-4f44-800f-a9850aed7772" (UID: "288ab83a-1d57-4f44-800f-a9850aed7772"). InnerVolumeSpecName "kube-api-access-z5qv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:18.506517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:18.478015    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288ab83a-1d57-4f44-800f-a9850aed7772-maas-api-xks-ca" pod "288ab83a-1d57-4f44-800f-a9850aed7772" (UID: "288ab83a-1d57-4f44-800f-a9850aed7772"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:18.581872 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:18.561662    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:22:18.603860 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:18.601465    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5qv2\" (UniqueName: \"kubernetes.io/projected/288ab83a-1d57-4f44-800f-a9850aed7772-kube-api-access-z5qv2\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:18.603860 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:18.601501    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288ab83a-1d57-4f44-800f-a9850aed7772-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:18.603860 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:18.601526    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288ab83a-1d57-4f44-800f-a9850aed7772-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:18.680854 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:18.674901    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:22:18.680854 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:18.674974    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894" podUID="de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:22:18.704137 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:18.704097    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:22:18.733374 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:18.717289    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2" podUID="28df0487-be29-4efe-99d6-94b6166fa1cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 16:22:18.733374 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:18.706740    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:19.322830 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:19.322591    6506 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.227s"
Nov 03 16:22:19.322830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.322687    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:22:19.322830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.322735    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:22:19.323145 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.322856    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:22:19.443875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.406360    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:19.443875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.406881    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:22:19.443875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.406931    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv" podUID="e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:22:19.641820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.595744    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:22:19.718565 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.691086    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:19.718565 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.692599    6506 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:19.718565 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.692654    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q" podUID="46df749f-6606-477d-85db-cfe8e35bd740" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:19.718565 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.693875    6506 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:22:19.718565 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.693933    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj" podUID="986e48d0-8cf9-45f3-86c7-06511006a2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:22:19.718565 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.697660    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2"
Nov 03 16:22:19.718565 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.713019    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 16:22:19.723821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.720779    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj" event={"ID":"288ab83a-1d57-4f44-800f-a9850aed7772","Type":"ContainerDied","Data":"b1a9963bce4e68818cd63f89c9d441248e2c7d0279f9d62904a0218595156a0f"}
Nov 03 16:22:19.723821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.720904    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1a9963bce4e68818cd63f89c9d441248e2c7d0279f9d62904a0218595156a0f"
Nov 03 16:22:19.723821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:19.721000    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"
Nov 03 16:22:20.240881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.212673    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:22:20.276403 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.275912    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:20.418881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.398989    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab2854b-0c08-4417-a5cf-bf67c191b51f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab2854b-0c08-4417-a5cf-bf67c191b51f-metrics-certs\") pod \"bab2854b-0c08-4417-a5cf-bf67c191b51f\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") "
Nov 03 16:22:20.418881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.399119    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab2854b-0c08-4417-a5cf-bf67c191b51f-kube-api-access-mtfh7\" (UniqueName: \"kubernetes.io/projected/bab2854b-0c08-4417-a5cf-bf67c191b51f-kube-api-access-mtfh7\") pod \"bab2854b-0c08-4417-a5cf-bf67c191b51f\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") "
Nov 03 16:22:20.418881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.399151    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab2854b-0c08-4417-a5cf-bf67c191b51f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab2854b-0c08-4417-a5cf-bf67c191b51f-maas-api-tls\") pod \"bab2854b-0c08-4417-a5cf-bf67c191b51f\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") "
Nov 03 16:22:20.418881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.399232    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab2854b-0c08-4417-a5cf-bf67c191b51f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab2854b-0c08-4417-a5cf-bf67c191b51f-openshift-service-ca\") pod \"bab2854b-0c08-4417-a5cf-bf67c191b51f\" (UID: \"bab2854b-0c08-4417-a5cf-bf67c191b51f\") "
Nov 03 16:22:20.433859 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.421160    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab2854b-0c08-4417-a5cf-bf67c191b51f-openshift-service-ca" pod "bab2854b-0c08-4417-a5cf-bf67c191b51f" (UID: "bab2854b-0c08-4417-a5cf-bf67c191b51f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:20.493885 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.489519    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:20.508836 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.508690    6506 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab2854b-0c08-4417-a5cf-bf67c191b51f-openshift-service-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:20.519559 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.519023    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab2854b-0c08-4417-a5cf-bf67c191b51f-metrics-certs" pod "bab2854b-0c08-4417-a5cf-bf67c191b51f" (UID: "bab2854b-0c08-4417-a5cf-bf67c191b51f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:20.527715 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.526353    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab2854b-0c08-4417-a5cf-bf67c191b51f-maas-api-tls" pod "bab2854b-0c08-4417-a5cf-bf67c191b51f" (UID: "bab2854b-0c08-4417-a5cf-bf67c191b51f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:20.607132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.604981    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:22:20.617553 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.617463    6506 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab2854b-0c08-4417-a5cf-bf67c191b51f-metrics-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:20.634333 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.563729    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab2854b-0c08-4417-a5cf-bf67c191b51f-kube-api-access-mtfh7" pod "bab2854b-0c08-4417-a5cf-bf67c191b51f" (UID: "bab2854b-0c08-4417-a5cf-bf67c191b51f"). InnerVolumeSpecName "kube-api-access-mtfh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:20.639944 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.639656    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:22:20.665904 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:20.665830    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ed082c_356f_45b5_80e5_c70a6f5d055d.slice/crio-conmon-f2136fcf41133c45299562b0c9bf9e4b86d61c40cb9f52092b95fd0a5ac329ac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46df749f_6606_477d_85db_cfe8e35bd740.slice/crio-conmon-2f514dd7a56e21bd532836d14728f4f63d0fc089fe38336ceb6fa1abb996ff42.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f7e7a1_02a1_4c19_9749_ce5ddaa44dba.slice/crio-bae9d4a98f92a04a924f676f8f05dc81b0a314e180b77203f67cd2ed32dc6bbc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46df749f_6606_477d_85db_cfe8e35bd740.slice/crio-2f514dd7a56e21bd532836d14728f4f63d0fc089fe38336ceb6fa1abb996ff42.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:22:20.667536 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:20.667362    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ed082c_356f_45b5_80e5_c70a6f5d055d.slice/crio-conmon-f2136fcf41133c45299562b0c9bf9e4b86d61c40cb9f52092b95fd0a5ac329ac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f7e7a1_02a1_4c19_9749_ce5ddaa44dba.slice/crio-bae9d4a98f92a04a924f676f8f05dc81b0a314e180b77203f67cd2ed32dc6bbc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f7e7a1_02a1_4c19_9749_ce5ddaa44dba.slice/crio-conmon-bae9d4a98f92a04a924f676f8f05dc81b0a314e180b77203f67cd2ed32dc6bbc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46df749f_6606_477d_85db_cfe8e35bd740.slice/crio-2f514dd7a56e21bd532836d14728f4f63d0fc089fe38336ceb6fa1abb996ff42.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46df749f_6606_477d_85db_cfe8e35bd740.slice/crio-conmon-2f514dd7a56e21bd532836d14728f4f63d0fc089fe38336ceb6fa1abb996ff42.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ed082c_356f_45b5_80e5_c70a6f5d055d.slice/crio-f2136fcf41133c45299562b0c9bf9e4b86d61c40cb9f52092b95fd0a5ac329ac.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:22:20.703975 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.703586    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:22:20.721225 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.720994    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtfh7\" (UniqueName: \"kubernetes.io/projected/bab2854b-0c08-4417-a5cf-bf67c191b51f-kube-api-access-mtfh7\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:20.721225 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.721028    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab2854b-0c08-4417-a5cf-bf67c191b51f-maas-api-tls\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:20.749260 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.749172    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw" event={"ID":"bab2854b-0c08-4417-a5cf-bf67c191b51f","Type":"ContainerDied","Data":"72ec86a0c5b8a5e3cca24c46cf089dd5b4ba1a26d3a4ea456323795d69c0eff6"}
Nov 03 16:22:20.749516 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.749491    6506 scope.go:122] "RemoveContainer" containerID="ccb5141d3367ac6dd7970f0f7be77e53af74b3dcc9416bc82779debfc54e6c6c"
Nov 03 16:22:20.749837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.749815    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"
Nov 03 16:22:20.759375 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.758536    6506 generic.go:356] "Generic (PLEG): container finished" podID="e0ed082c-356f-45b5-80e5-c70a6f5d055d" containerID="f2136fcf41133c45299562b0c9bf9e4b86d61c40cb9f52092b95fd0a5ac329ac" exitCode=0
Nov 03 16:22:20.762454 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.762429    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5" event={"ID":"e0ed082c-356f-45b5-80e5-c70a6f5d055d","Type":"ContainerDied","Data":"f2136fcf41133c45299562b0c9bf9e4b86d61c40cb9f52092b95fd0a5ac329ac"}
Nov 03 16:22:20.764730 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.764704    6506 generic.go:356] "Generic (PLEG): container finished" podID="39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" containerID="bae9d4a98f92a04a924f676f8f05dc81b0a314e180b77203f67cd2ed32dc6bbc" exitCode=0
Nov 03 16:22:20.765022 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.764992    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w" event={"ID":"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba","Type":"ContainerDied","Data":"bae9d4a98f92a04a924f676f8f05dc81b0a314e180b77203f67cd2ed32dc6bbc"}
Nov 03 16:22:20.769518 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.769494    6506 generic.go:356] "Generic (PLEG): container finished" podID="46df749f-6606-477d-85db-cfe8e35bd740" containerID="2f514dd7a56e21bd532836d14728f4f63d0fc089fe38336ceb6fa1abb996ff42" exitCode=0
Nov 03 16:22:20.770529 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.770506    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q" event={"ID":"46df749f-6606-477d-85db-cfe8e35bd740","Type":"ContainerDied","Data":"2f514dd7a56e21bd532836d14728f4f63d0fc089fe38336ceb6fa1abb996ff42"}
Nov 03 16:22:20.775546 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.775528    6506 scope.go:122] "RemoveContainer" containerID="36ae5eab43b7543917c52fcfb1337e71d5e7c819767dac31f60642adb1f4e97b"
Nov 03 16:22:20.807467 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.807441    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"]
Nov 03 16:22:20.823843 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.816684    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:22:20.831018 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.830990    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-afb990-66b4dd9956-857bw"]
Nov 03 16:22:20.872850 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.872752    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz" podUID="39813549-9eb7-4cb7-9f08-4bf983b4cebc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:22:20.948928 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.943513    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-token\" (UniqueName: \"kubernetes.io/projected/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-token\") pod \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") "
Nov 03 16:22:20.948928 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.947635    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istiod-ca-cert\") pod \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") "
Nov 03 16:22:20.948928 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.947749    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-credential-socket\") pod \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") "
Nov 03 16:22:20.948928 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.947851    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-podinfo\") pod \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") "
Nov 03 16:22:20.948928 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.947885    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-envoy\") pod \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") "
Nov 03 16:22:20.948928 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.947959    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-data\") pod \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") "
Nov 03 16:22:20.948928 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.948093    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-workload-socket\") pod \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") "
Nov 03 16:22:20.948928 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.948131    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ed082c-356f-45b5-80e5-c70a6f5d055d-kube-api-access-rcsrb\" (UniqueName: \"kubernetes.io/projected/e0ed082c-356f-45b5-80e5-c70a6f5d055d-kube-api-access-rcsrb\") pod \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") "
Nov 03 16:22:20.948928 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.948172    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-workload-certs\") pod \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\" (UID: \"e0ed082c-356f-45b5-80e5-c70a6f5d055d\") "
Nov 03 16:22:20.951416 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.951186    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istiod-ca-cert" pod "e0ed082c-356f-45b5-80e5-c70a6f5d055d" (UID: "e0ed082c-356f-45b5-80e5-c70a6f5d055d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:20.952749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.951586    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-credential-socket" pod "e0ed082c-356f-45b5-80e5-c70a6f5d055d" (UID: "e0ed082c-356f-45b5-80e5-c70a6f5d055d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:20.952749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.952464    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-data" pod "e0ed082c-356f-45b5-80e5-c70a6f5d055d" (UID: "e0ed082c-356f-45b5-80e5-c70a6f5d055d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:20.952749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.952566    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-workload-socket" pod "e0ed082c-356f-45b5-80e5-c70a6f5d055d" (UID: "e0ed082c-356f-45b5-80e5-c70a6f5d055d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:20.953218 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.953169    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-workload-certs" pod "e0ed082c-356f-45b5-80e5-c70a6f5d055d" (UID: "e0ed082c-356f-45b5-80e5-c70a6f5d055d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:20.960240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.958960    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-envoy" pod "e0ed082c-356f-45b5-80e5-c70a6f5d055d" (UID: "e0ed082c-356f-45b5-80e5-c70a6f5d055d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:20.960240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.959078    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ed082c-356f-45b5-80e5-c70a6f5d055d-kube-api-access-rcsrb" pod "e0ed082c-356f-45b5-80e5-c70a6f5d055d" (UID: "e0ed082c-356f-45b5-80e5-c70a6f5d055d"). InnerVolumeSpecName "kube-api-access-rcsrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:20.960240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.959158    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-token" pod "e0ed082c-356f-45b5-80e5-c70a6f5d055d" (UID: "e0ed082c-356f-45b5-80e5-c70a6f5d055d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:20.960240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.959242    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-podinfo" pod "e0ed082c-356f-45b5-80e5-c70a6f5d055d" (UID: "e0ed082c-356f-45b5-80e5-c70a6f5d055d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:20.985009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.984979    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:22:20.988069 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:20.988032    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:22:21.049743 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.049711    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.049743 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.049745    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcsrb\" (UniqueName: \"kubernetes.io/projected/e0ed082c-356f-45b5-80e5-c70a6f5d055d-kube-api-access-rcsrb\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.050032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.049761    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.050032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.049776    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.050032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.049789    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.050032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.049821    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.050032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.049835    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.050032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.049848    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.050032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.049863    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ed082c-356f-45b5-80e5-c70a6f5d055d-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.151107 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151066    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-data\") pod \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") "
Nov 03 16:22:21.151279 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151160    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-credential-socket\") pod \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") "
Nov 03 16:22:21.151279 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151252    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46df749f-6606-477d-85db-cfe8e35bd740-istio-token\" (UniqueName: \"kubernetes.io/projected/46df749f-6606-477d-85db-cfe8e35bd740-istio-token\") pod \"46df749f-6606-477d-85db-cfe8e35bd740\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") "
Nov 03 16:22:21.151369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151284    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-envoy\") pod \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") "
Nov 03 16:22:21.151369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151320    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-workload-socket\") pod \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") "
Nov 03 16:22:21.151369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151350    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46df749f-6606-477d-85db-cfe8e35bd740-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46df749f-6606-477d-85db-cfe8e35bd740-istiod-ca-cert\") pod \"46df749f-6606-477d-85db-cfe8e35bd740\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") "
Nov 03 16:22:21.151501 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151364    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-data" pod "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" (UID: "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.151501 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151396    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-token\" (UniqueName: \"kubernetes.io/projected/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-token\") pod \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") "
Nov 03 16:22:21.151501 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151457    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-credential-socket" pod "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" (UID: "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.151501 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151482    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-istio-data\") pod \"46df749f-6606-477d-85db-cfe8e35bd740\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") "
Nov 03 16:22:21.151690 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151532    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-istio-envoy\") pod \"46df749f-6606-477d-85db-cfe8e35bd740\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") "
Nov 03 16:22:21.151690 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151563    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-workload-certs\") pod \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") "
Nov 03 16:22:21.151690 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151589    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46df749f-6606-477d-85db-cfe8e35bd740-kube-api-access-hsdh8\" (UniqueName: \"kubernetes.io/projected/46df749f-6606-477d-85db-cfe8e35bd740-kube-api-access-hsdh8\") pod \"46df749f-6606-477d-85db-cfe8e35bd740\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") "
Nov 03 16:22:21.151690 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151624    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-kube-api-access-qqqh5\" (UniqueName: \"kubernetes.io/projected/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-kube-api-access-qqqh5\") pod \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") "
Nov 03 16:22:21.151690 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151627    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-workload-socket" pod "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" (UID: "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.151690 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151682    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46df749f-6606-477d-85db-cfe8e35bd740-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46df749f-6606-477d-85db-cfe8e35bd740-istio-podinfo\") pod \"46df749f-6606-477d-85db-cfe8e35bd740\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") "
Nov 03 16:22:21.152038 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151748    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-podinfo\") pod \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") "
Nov 03 16:22:21.152038 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151820    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-workload-certs\") pod \"46df749f-6606-477d-85db-cfe8e35bd740\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") "
Nov 03 16:22:21.152038 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151864    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istiod-ca-cert\") pod \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\" (UID: \"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba\") "
Nov 03 16:22:21.152038 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151867    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-istio-data" pod "46df749f-6606-477d-85db-cfe8e35bd740" (UID: "46df749f-6606-477d-85db-cfe8e35bd740"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.152038 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151902    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46df749f-6606-477d-85db-cfe8e35bd740-istiod-ca-cert" pod "46df749f-6606-477d-85db-cfe8e35bd740" (UID: "46df749f-6606-477d-85db-cfe8e35bd740"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:21.152038 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151930    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-workload-socket\") pod \"46df749f-6606-477d-85db-cfe8e35bd740\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") "
Nov 03 16:22:21.152038 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.151992    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-credential-socket\") pod \"46df749f-6606-477d-85db-cfe8e35bd740\" (UID: \"46df749f-6606-477d-85db-cfe8e35bd740\") "
Nov 03 16:22:21.152382 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.152299    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-workload-certs" pod "46df749f-6606-477d-85db-cfe8e35bd740" (UID: "46df749f-6606-477d-85db-cfe8e35bd740"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.152506 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.152480    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-workload-certs" pod "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" (UID: "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.152705 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.152682    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-credential-socket" pod "46df749f-6606-477d-85db-cfe8e35bd740" (UID: "46df749f-6606-477d-85db-cfe8e35bd740"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.152913 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.152893    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.153001 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.152917    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.153001 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.152933    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.153001 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.152947    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46df749f-6606-477d-85db-cfe8e35bd740-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.153001 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.152961    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.153001 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.152975    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.153001 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.152990    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.153338 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.153004    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.153414 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.153382    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istiod-ca-cert" pod "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" (UID: "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:21.153705 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.153621    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-workload-socket" pod "46df749f-6606-477d-85db-cfe8e35bd740" (UID: "46df749f-6606-477d-85db-cfe8e35bd740"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.154346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.154286    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-podinfo" pod "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" (UID: "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:21.154611 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.154539    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-envoy" pod "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" (UID: "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.154611 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.154594    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46df749f-6606-477d-85db-cfe8e35bd740-istio-token" pod "46df749f-6606-477d-85db-cfe8e35bd740" (UID: "46df749f-6606-477d-85db-cfe8e35bd740"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:21.155343 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.155320    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-kube-api-access-qqqh5" pod "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" (UID: "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba"). InnerVolumeSpecName "kube-api-access-qqqh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:21.155612 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.155591    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-istio-envoy" pod "46df749f-6606-477d-85db-cfe8e35bd740" (UID: "46df749f-6606-477d-85db-cfe8e35bd740"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.156071 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.156048    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46df749f-6606-477d-85db-cfe8e35bd740-istio-podinfo" pod "46df749f-6606-477d-85db-cfe8e35bd740" (UID: "46df749f-6606-477d-85db-cfe8e35bd740"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:21.156298 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.156264    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46df749f-6606-477d-85db-cfe8e35bd740-kube-api-access-hsdh8" pod "46df749f-6606-477d-85db-cfe8e35bd740" (UID: "46df749f-6606-477d-85db-cfe8e35bd740"). InnerVolumeSpecName "kube-api-access-hsdh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:21.156570 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.156552    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-token" pod "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" (UID: "39f7e7a1-02a1-4c19-9749-ce5ddaa44dba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:21.254326 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.254289    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.254326 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.254321    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.254326 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.254330    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46df749f-6606-477d-85db-cfe8e35bd740-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.254960 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.254339    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.254960 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.254348    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.254960 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.254356    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46df749f-6606-477d-85db-cfe8e35bd740-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.254960 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.254364    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsdh8\" (UniqueName: \"kubernetes.io/projected/46df749f-6606-477d-85db-cfe8e35bd740-kube-api-access-hsdh8\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.254960 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.254374    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqqh5\" (UniqueName: \"kubernetes.io/projected/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-kube-api-access-qqqh5\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.254960 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.254383    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46df749f-6606-477d-85db-cfe8e35bd740-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.254960 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.254392    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.559141 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.559093    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" containerID="cri-o://0d05bfa15953053ae1a9eff19166c28a92848cc2c0092f99a618ccf142f15b5d" gracePeriod=45
Nov 03 16:22:21.634220 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.634181    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"]
Nov 03 16:22:21.670249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.670205    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": dial tcp 10.133.0.60:8000: connect: connection refused"
Nov 03 16:22:21.790442 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.790404    6506 generic.go:356] "Generic (PLEG): container finished" podID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerID="0d05bfa15953053ae1a9eff19166c28a92848cc2c0092f99a618ccf142f15b5d" exitCode=0
Nov 03 16:22:21.790627 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.790540    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" event={"ID":"3bd55dad-fd00-47aa-84a0-3c38d7be189f","Type":"ContainerDied","Data":"0d05bfa15953053ae1a9eff19166c28a92848cc2c0092f99a618ccf142f15b5d"}
Nov 03 16:22:21.792454 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.792422    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q" event={"ID":"46df749f-6606-477d-85db-cfe8e35bd740","Type":"ContainerDied","Data":"9f0619ffaf3d9f6e3be14f0ab7e5434d266e575d643b4f93065bd94b60c84571"}
Nov 03 16:22:21.792606 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.792488    6506 scope.go:122] "RemoveContainer" containerID="2f514dd7a56e21bd532836d14728f4f63d0fc089fe38336ceb6fa1abb996ff42"
Nov 03 16:22:21.792675 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.792632    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"
Nov 03 16:22:21.804643 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.804622    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"
Nov 03 16:22:21.805155 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.804618    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5" event={"ID":"e0ed082c-356f-45b5-80e5-c70a6f5d055d","Type":"ContainerDied","Data":"c08e16f6662088843697aa6821be69405c4bf0376d589e293135553382f99d1d"}
Nov 03 16:22:21.806544 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.806506    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w" event={"ID":"39f7e7a1-02a1-4c19-9749-ce5ddaa44dba","Type":"ContainerDied","Data":"d0bac3761872a3ab7ebc04e9e764539e7a9052d150d3477d25b053a6f3fcb88c"}
Nov 03 16:22:21.809079 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.808641    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"
Nov 03 16:22:21.834656 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.834625    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"]
Nov 03 16:22:21.835198 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.834788    6506 scope.go:122] "RemoveContainer" containerID="f2136fcf41133c45299562b0c9bf9e4b86d61c40cb9f52092b95fd0a5ac329ac"
Nov 03 16:22:21.835198 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.834964    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:21.836360 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.836336    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:22:21.843177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.843151    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-2r29q"]
Nov 03 16:22:21.848376 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.848356    6506 scope.go:122] "RemoveContainer" containerID="bae9d4a98f92a04a924f676f8f05dc81b0a314e180b77203f67cd2ed32dc6bbc"
Nov 03 16:22:21.929844 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.929808    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"]
Nov 03 16:22:21.933589 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.933562    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-k798w"]
Nov 03 16:22:21.957666 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.957634    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"]
Nov 03 16:22:21.964054 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.964014    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-home\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-home\") pod \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") "
Nov 03 16:22:21.964192 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.964142    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bd55dad-fd00-47aa-84a0-3c38d7be189f-tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd55dad-fd00-47aa-84a0-3c38d7be189f-tls-certs\") pod \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") "
Nov 03 16:22:21.964192 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.964175    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-model-cache\") pod \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") "
Nov 03 16:22:21.964292 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.964199    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-tmp-dir\") pod \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") "
Nov 03 16:22:21.964292 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.964283    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-dshm\") pod \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") "
Nov 03 16:22:21.964354 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.964340    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd55dad-fd00-47aa-84a0-3c38d7be189f-kube-api-access-v8p6k\" (UniqueName: \"kubernetes.io/projected/3bd55dad-fd00-47aa-84a0-3c38d7be189f-kube-api-access-v8p6k\") pod \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\" (UID: \"3bd55dad-fd00-47aa-84a0-3c38d7be189f\") "
Nov 03 16:22:21.965282 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.965247    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-home" pod "3bd55dad-fd00-47aa-84a0-3c38d7be189f" (UID: "3bd55dad-fd00-47aa-84a0-3c38d7be189f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.965282 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.965257    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-model-cache" pod "3bd55dad-fd00-47aa-84a0-3c38d7be189f" (UID: "3bd55dad-fd00-47aa-84a0-3c38d7be189f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.965467 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.965363    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-tmp-dir" pod "3bd55dad-fd00-47aa-84a0-3c38d7be189f" (UID: "3bd55dad-fd00-47aa-84a0-3c38d7be189f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.967420 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.967390    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd55dad-fd00-47aa-84a0-3c38d7be189f-kube-api-access-v8p6k" pod "3bd55dad-fd00-47aa-84a0-3c38d7be189f" (UID: "3bd55dad-fd00-47aa-84a0-3c38d7be189f"). InnerVolumeSpecName "kube-api-access-v8p6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:21.967851 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.967828    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-dshm" pod "3bd55dad-fd00-47aa-84a0-3c38d7be189f" (UID: "3bd55dad-fd00-47aa-84a0-3c38d7be189f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.967971 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.967852    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd55dad-fd00-47aa-84a0-3c38d7be189f-tls-certs" pod "3bd55dad-fd00-47aa-84a0-3c38d7be189f" (UID: "3bd55dad-fd00-47aa-84a0-3c38d7be189f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:21.969207 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:21.969185    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-5sxg5"]
Nov 03 16:22:22.057975 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.057934    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:22.065372 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.065343    6506 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-dshm\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:22.065372 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.065366    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8p6k\" (UniqueName: \"kubernetes.io/projected/3bd55dad-fd00-47aa-84a0-3c38d7be189f-kube-api-access-v8p6k\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:22.065372 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.065377    6506 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-home\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:22.065650 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.065386    6506 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd55dad-fd00-47aa-84a0-3c38d7be189f-tls-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:22.065650 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.065395    6506 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-model-cache\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:22.065650 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.065404    6506 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd55dad-fd00-47aa-84a0-3c38d7be189f-tmp-dir\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:22.107325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.107250    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" path="/var/lib/kubelet/pods/39f7e7a1-02a1-4c19-9749-ce5ddaa44dba/volumes"
Nov 03 16:22:22.108080 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.108055    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46df749f-6606-477d-85db-cfe8e35bd740" path="/var/lib/kubelet/pods/46df749f-6606-477d-85db-cfe8e35bd740/volumes"
Nov 03 16:22:22.108999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.108961    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab2854b-0c08-4417-a5cf-bf67c191b51f" path="/var/lib/kubelet/pods/bab2854b-0c08-4417-a5cf-bf67c191b51f/volumes"
Nov 03 16:22:22.110067 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.110048    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ed082c-356f-45b5-80e5-c70a6f5d055d" path="/var/lib/kubelet/pods/e0ed082c-356f-45b5-80e5-c70a6f5d055d/volumes"
Nov 03 16:22:22.702528 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.702450    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:22.824072 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.824040    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf" event={"ID":"3bd55dad-fd00-47aa-84a0-3c38d7be189f","Type":"ContainerDied","Data":"0acc0ed53e4a84b0cbdbb6f3e622d5799f20bf7a77a27bc397b223c103c516e0"}
Nov 03 16:22:22.824072 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.824063    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"
Nov 03 16:22:22.824333 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.824095    6506 scope.go:122] "RemoveContainer" containerID="0d05bfa15953053ae1a9eff19166c28a92848cc2c0092f99a618ccf142f15b5d"
Nov 03 16:22:22.844778 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.844742    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"]
Nov 03 16:22:22.847954 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:22.847928    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/premium-w4-4a19cd-kserve-9f6896644-7b6vf"]
Nov 03 16:22:23.368311 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:23.368277    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:23.741740 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:23.741637    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:24.112817 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:24.112126    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" path="/var/lib/kubelet/pods/3bd55dad-fd00-47aa-84a0-3c38d7be189f/volumes"
Nov 03 16:22:24.758095 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:24.757991    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:25.105651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:25.105619    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:25.307915 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:25.307884    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"]
Nov 03 16:22:25.308114 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:25.308078    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j" podUID="af0fa8f4-44df-45e7-8028-7d329877afaa" containerName="istio-proxy" containerID="cri-o://036dad6f01f50d4737e26f3850a56715895fa42e162ae6a7b82cd0d405d6809d" gracePeriod=30
Nov 03 16:22:25.332990 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:25.332964    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"]
Nov 03 16:22:25.333202 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:25.333182    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv" podUID="e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" containerName="istio-proxy" containerID="cri-o://e636f1aec15879049649c5f555ae75b91d9e3b5727cbf2971acd5fa5eb0ad2ae" gracePeriod=30
Nov 03 16:22:25.337927 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:25.337898    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:25.339867 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:25.339848    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"]
Nov 03 16:22:25.340028 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:25.340012    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj" podUID="986e48d0-8cf9-45f3-86c7-06511006a2e1" containerName="istio-proxy" containerID="cri-o://28e7f6ac50570950bd5ce77ffaada26044e4ce5d4cddea9c89db7d8fc0127025" gracePeriod=30
Nov 03 16:22:25.534261 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:25.534220    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:25.720250 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:25.720213    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:26.456917 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:26.456872    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:27.281408 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:27.281376    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"]
Nov 03 16:22:27.310286 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:27.310257    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"]
Nov 03 16:22:27.443205 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:27.443172    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:27.628883 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:27.620578    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:28.146346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146314    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"]
Nov 03 16:22:28.146820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146786    6506 cpu_manager.go:414] "containerMap: removing container" podUID="e0ed082c-356f-45b5-80e5-c70a6f5d055d" containerName="istio-proxy"
Nov 03 16:22:28.146820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146817    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ed082c-356f-45b5-80e5-c70a6f5d055d" containerName="istio-proxy"
Nov 03 16:22:28.146922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146827    6506 cpu_manager.go:414] "containerMap: removing container" podUID="288ab83a-1d57-4f44-800f-a9850aed7772" containerName="revoke-keys"
Nov 03 16:22:28.146922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146832    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288ab83a-1d57-4f44-800f-a9850aed7772" containerName="revoke-keys"
Nov 03 16:22:28.146922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146848    6506 cpu_manager.go:414] "containerMap: removing container" podUID="e0ed082c-356f-45b5-80e5-c70a6f5d055d" containerName="istio-proxy"
Nov 03 16:22:28.146922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146853    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ed082c-356f-45b5-80e5-c70a6f5d055d" containerName="istio-proxy"
Nov 03 16:22:28.146922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146861    6506 cpu_manager.go:414] "containerMap: removing container" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main"
Nov 03 16:22:28.146922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146865    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main"
Nov 03 16:22:28.146922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146884    6506 cpu_manager.go:414] "containerMap: removing container" podUID="bab2854b-0c08-4417-a5cf-bf67c191b51f" containerName="maas-api"
Nov 03 16:22:28.146922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146888    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab2854b-0c08-4417-a5cf-bf67c191b51f" containerName="maas-api"
Nov 03 16:22:28.146922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146903    6506 cpu_manager.go:414] "containerMap: removing container" podUID="39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" containerName="istio-proxy"
Nov 03 16:22:28.146922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146907    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" containerName="istio-proxy"
Nov 03 16:22:28.146922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146915    6506 cpu_manager.go:414] "containerMap: removing container" podUID="46df749f-6606-477d-85db-cfe8e35bd740" containerName="istio-proxy"
Nov 03 16:22:28.146922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146919    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46df749f-6606-477d-85db-cfe8e35bd740" containerName="istio-proxy"
Nov 03 16:22:28.147255 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.146995    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f7e7a1-02a1-4c19-9749-ce5ddaa44dba" containerName="istio-proxy"
Nov 03 16:22:28.147255 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.147006    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="46df749f-6606-477d-85db-cfe8e35bd740" containerName="istio-proxy"
Nov 03 16:22:28.147255 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.147014    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ed082c-356f-45b5-80e5-c70a6f5d055d" containerName="istio-proxy"
Nov 03 16:22:28.147255 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.147021    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="288ab83a-1d57-4f44-800f-a9850aed7772" containerName="revoke-keys"
Nov 03 16:22:28.147255 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.147028    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ed082c-356f-45b5-80e5-c70a6f5d055d" containerName="istio-proxy"
Nov 03 16:22:28.147255 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.147033    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd55dad-fd00-47aa-84a0-3c38d7be189f" containerName="main"
Nov 03 16:22:28.147255 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.147041    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab2854b-0c08-4417-a5cf-bf67c191b51f" containerName="maas-api"
Nov 03 16:22:28.151927 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.151911    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"
Nov 03 16:22:28.156788 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.156767    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:22:28.159179 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.159160    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:22:28.159351 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.159337    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:22:28.171608 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.171587    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"]
Nov 03 16:22:28.203076 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.203051    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f093183a-3336-4dce-8945-f907211722f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d89707d-lh546\" (UID: \"f093183a-3336-4dce-8945-f907211722f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"
Nov 03 16:22:28.203210 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.203119    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f093183a-3336-4dce-8945-f907211722f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d89707d-lh546\" (UID: \"f093183a-3336-4dce-8945-f907211722f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"
Nov 03 16:22:28.203337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.203315    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c42k\" (UniqueName: \"kubernetes.io/projected/f093183a-3336-4dce-8945-f907211722f4-kube-api-access-2c42k\") pod \"maas-api-revoke-sub-5d89707d-lh546\" (UID: \"f093183a-3336-4dce-8945-f907211722f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"
Nov 03 16:22:28.257921 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.257897    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:22:28.304698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.304670    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f093183a-3336-4dce-8945-f907211722f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d89707d-lh546\" (UID: \"f093183a-3336-4dce-8945-f907211722f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"
Nov 03 16:22:28.304845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.304830    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c42k\" (UniqueName: \"kubernetes.io/projected/f093183a-3336-4dce-8945-f907211722f4-kube-api-access-2c42k\") pod \"maas-api-revoke-sub-5d89707d-lh546\" (UID: \"f093183a-3336-4dce-8945-f907211722f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"
Nov 03 16:22:28.304912 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.304891    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f093183a-3336-4dce-8945-f907211722f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d89707d-lh546\" (UID: \"f093183a-3336-4dce-8945-f907211722f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"
Nov 03 16:22:28.305533 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.305505    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f093183a-3336-4dce-8945-f907211722f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d89707d-lh546\" (UID: \"f093183a-3336-4dce-8945-f907211722f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"
Nov 03 16:22:28.307156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.307125    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f093183a-3336-4dce-8945-f907211722f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d89707d-lh546\" (UID: \"f093183a-3336-4dce-8945-f907211722f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"
Nov 03 16:22:28.318101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.318080    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c42k\" (UniqueName: \"kubernetes.io/projected/f093183a-3336-4dce-8945-f907211722f4-kube-api-access-2c42k\") pod \"maas-api-revoke-sub-5d89707d-lh546\" (UID: \"f093183a-3336-4dce-8945-f907211722f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"
Nov 03 16:22:28.465754 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.465676    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"
Nov 03 16:22:28.485395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.485366    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:28.663631 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.663592    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:28.753432 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.753351    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"]
Nov 03 16:22:28.760190 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:22:28.760142    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf093183a_3336_4dce_8945_f907211722f4.slice/crio-d1b5d0b8f87033ac24b955f93cf0eff85b683629a71e9460c5713e42e4c00e6e WatchSource:0}: Error finding container d1b5d0b8f87033ac24b955f93cf0eff85b683629a71e9460c5713e42e4c00e6e: Status 404 returned error can't find the container with id d1b5d0b8f87033ac24b955f93cf0eff85b683629a71e9460c5713e42e4c00e6e
Nov 03 16:22:28.829842 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.824836    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"]
Nov 03 16:22:28.830822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.830413    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2" podUID="5a76c7da-5ade-41c8-a075-430ccf3e7215" containerName="istio-proxy" containerID="cri-o://90b4c690b6dd52b9577f008d24d2b39d7c076fa2fd75e0a421c73138c6cbbe4c" gracePeriod=30
Nov 03 16:22:28.835017 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.834652    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"]
Nov 03 16:22:28.835017 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.834878    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml" podUID="119e87d8-1b7b-46db-9ac8-289804f75184" containerName="istio-proxy" containerID="cri-o://1e606cb3a3438c6c9d3240e6ea43b8da8665f969a53943b64aa7ce083b54d288" gracePeriod=30
Nov 03 16:22:28.840524 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.839662    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"]
Nov 03 16:22:28.840524 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.839895    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556" podUID="17b47389-9b79-402f-bc9b-d404e0b1e75c" containerName="istio-proxy" containerID="cri-o://b46da3dbf1aa72f92e8acec9d704c7c5b0d23b8513f3038f0e8e0eadfc51dc57" gracePeriod=30
Nov 03 16:22:28.840524 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:28.840410    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:22:28.844915 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.844896    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"]
Nov 03 16:22:28.845097 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.845079    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw" podUID="20b9b08d-2d97-4604-890c-10f6305c0108" containerName="istio-proxy" containerID="cri-o://7234d9df4269a85af5e0ed9fe525aaa89ea6ed11dbe23c25ddeec25105b25298" gracePeriod=30
Nov 03 16:22:28.870634 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.870610    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546" event={"ID":"f093183a-3336-4dce-8945-f907211722f4","Type":"ContainerStarted","Data":"d1b5d0b8f87033ac24b955f93cf0eff85b683629a71e9460c5713e42e4c00e6e"}
Nov 03 16:22:28.926870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:28.919153    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:29.331700 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:29.331598    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:29.467826 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:29.467766    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:29.467990 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:29.467845    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw" podUID="20b9b08d-2d97-4604-890c-10f6305c0108" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:29.620867 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:29.620766    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:29.830012 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:29.829982    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:29.879590 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:29.879517    6506 generic.go:356] "Generic (PLEG): container finished" podID="f093183a-3336-4dce-8945-f907211722f4" containerID="c5fdfc94540a8f62b34a3e2196b3726b8deb92735288a2e1d5785abf888bc118" exitCode=0
Nov 03 16:22:29.879590 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:29.879569    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546" event={"ID":"f093183a-3336-4dce-8945-f907211722f4","Type":"ContainerDied","Data":"c5fdfc94540a8f62b34a3e2196b3726b8deb92735288a2e1d5785abf888bc118"}
Nov 03 16:22:30.211819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.211706    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:30.436230 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.436188    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:30.624765 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.624735    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:22:30.673186 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.673164    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:22:30.675901 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.675881    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:22:30.772578 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.772546    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-envoy\") pod \"af0fa8f4-44df-45e7-8028-7d329877afaa\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") "
Nov 03 16:22:30.772730 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.772598    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-podinfo\") pod \"986e48d0-8cf9-45f3-86c7-06511006a2e1\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") "
Nov 03 16:22:30.772730 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.772633    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-workload-certs\") pod \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") "
Nov 03 16:22:30.772730 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.772676    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-data\") pod \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") "
Nov 03 16:22:30.772730 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.772702    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-envoy\") pod \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") "
Nov 03 16:22:30.773036 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.772849    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-token\" (UniqueName: \"kubernetes.io/projected/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-token\") pod \"af0fa8f4-44df-45e7-8028-7d329877afaa\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") "
Nov 03 16:22:30.773036 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.772899    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-credential-socket\") pod \"af0fa8f4-44df-45e7-8028-7d329877afaa\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") "
Nov 03 16:22:30.773036 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.772976    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-data\") pod \"af0fa8f4-44df-45e7-8028-7d329877afaa\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") "
Nov 03 16:22:30.773036 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.772993    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-workload-certs" pod "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" (UID: "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.773036 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.773011    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-workload-socket\") pod \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") "
Nov 03 16:22:30.773036 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.773010    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-data" pod "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" (UID: "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.773357 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.773046    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-envoy\") pod \"986e48d0-8cf9-45f3-86c7-06511006a2e1\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") "
Nov 03 16:22:30.773357 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.773092    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0fa8f4-44df-45e7-8028-7d329877afaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0fa8f4-44df-45e7-8028-7d329877afaa-istiod-ca-cert\") pod \"af0fa8f4-44df-45e7-8028-7d329877afaa\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") "
Nov 03 16:22:30.773357 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.773135    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-token\" (UniqueName: \"kubernetes.io/projected/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-token\") pod \"986e48d0-8cf9-45f3-86c7-06511006a2e1\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") "
Nov 03 16:22:30.773357 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.773191    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0fa8f4-44df-45e7-8028-7d329877afaa-kube-api-access-8gnvn\" (UniqueName: \"kubernetes.io/projected/af0fa8f4-44df-45e7-8028-7d329877afaa-kube-api-access-8gnvn\") pod \"af0fa8f4-44df-45e7-8028-7d329877afaa\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") "
Nov 03 16:22:30.773357 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.773224    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-podinfo\") pod \"af0fa8f4-44df-45e7-8028-7d329877afaa\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") "
Nov 03 16:22:30.773357 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.773264    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-token\" (UniqueName: \"kubernetes.io/projected/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-token\") pod \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") "
Nov 03 16:22:30.773357 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.773276    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-data" pod "af0fa8f4-44df-45e7-8028-7d329877afaa" (UID: "af0fa8f4-44df-45e7-8028-7d329877afaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.773357 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.773289    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/986e48d0-8cf9-45f3-86c7-06511006a2e1-kube-api-access-tsp6l\" (UniqueName: \"kubernetes.io/projected/986e48d0-8cf9-45f3-86c7-06511006a2e1-kube-api-access-tsp6l\") pod \"986e48d0-8cf9-45f3-86c7-06511006a2e1\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") "
Nov 03 16:22:30.773357 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.773289    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-credential-socket" pod "af0fa8f4-44df-45e7-8028-7d329877afaa" (UID: "af0fa8f4-44df-45e7-8028-7d329877afaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.777101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.777072    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-envoy" pod "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" (UID: "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.778087 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.777723    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0fa8f4-44df-45e7-8028-7d329877afaa-kube-api-access-8gnvn" pod "af0fa8f4-44df-45e7-8028-7d329877afaa" (UID: "af0fa8f4-44df-45e7-8028-7d329877afaa"). InnerVolumeSpecName "kube-api-access-8gnvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:30.778087 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.777769    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-envoy" pod "af0fa8f4-44df-45e7-8028-7d329877afaa" (UID: "af0fa8f4-44df-45e7-8028-7d329877afaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.778087 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.778016    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-token" pod "af0fa8f4-44df-45e7-8028-7d329877afaa" (UID: "af0fa8f4-44df-45e7-8028-7d329877afaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:30.778087 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.778048    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-podinfo" pod "986e48d0-8cf9-45f3-86c7-06511006a2e1" (UID: "986e48d0-8cf9-45f3-86c7-06511006a2e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:30.778396 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.778096    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-podinfo\") pod \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") "
Nov 03 16:22:30.778396 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.778139    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/986e48d0-8cf9-45f3-86c7-06511006a2e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/986e48d0-8cf9-45f3-86c7-06511006a2e1-istiod-ca-cert\") pod \"986e48d0-8cf9-45f3-86c7-06511006a2e1\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") "
Nov 03 16:22:30.778396 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.778172    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-credential-socket\") pod \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") "
Nov 03 16:22:30.778396 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.778204    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-credential-socket\") pod \"986e48d0-8cf9-45f3-86c7-06511006a2e1\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") "
Nov 03 16:22:30.778396 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.778236    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-workload-socket\") pod \"af0fa8f4-44df-45e7-8028-7d329877afaa\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") "
Nov 03 16:22:30.778396 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.778245    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-workload-socket" pod "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" (UID: "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.778396 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.778283    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-data\") pod \"986e48d0-8cf9-45f3-86c7-06511006a2e1\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") "
Nov 03 16:22:30.778396 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.778343    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-workload-certs\") pod \"986e48d0-8cf9-45f3-86c7-06511006a2e1\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") "
Nov 03 16:22:30.778396 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.778376    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-workload-certs\") pod \"af0fa8f4-44df-45e7-8028-7d329877afaa\" (UID: \"af0fa8f4-44df-45e7-8028-7d329877afaa\") "
Nov 03 16:22:30.779126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.778412    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-kube-api-access-mq68p\" (UniqueName: \"kubernetes.io/projected/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-kube-api-access-mq68p\") pod \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") "
Nov 03 16:22:30.779126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.778524    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istiod-ca-cert\") pod \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\" (UID: \"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3\") "
Nov 03 16:22:30.779126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.778549    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986e48d0-8cf9-45f3-86c7-06511006a2e1-istiod-ca-cert" pod "986e48d0-8cf9-45f3-86c7-06511006a2e1" (UID: "986e48d0-8cf9-45f3-86c7-06511006a2e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:30.779126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.778575    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-workload-socket\") pod \"986e48d0-8cf9-45f3-86c7-06511006a2e1\" (UID: \"986e48d0-8cf9-45f3-86c7-06511006a2e1\") "
Nov 03 16:22:30.779342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.779223    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.779342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.779245    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.779342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.779259    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gnvn\" (UniqueName: \"kubernetes.io/projected/af0fa8f4-44df-45e7-8028-7d329877afaa-kube-api-access-8gnvn\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.779342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.779273    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/986e48d0-8cf9-45f3-86c7-06511006a2e1-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.779342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.779285    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.779342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.779298    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.779342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.779310    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.779342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.779321    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.779342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.779333    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.779342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.779345    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.779902 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.779358    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.779902 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.779665    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-workload-socket" pod "986e48d0-8cf9-45f3-86c7-06511006a2e1" (UID: "986e48d0-8cf9-45f3-86c7-06511006a2e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.779902 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.779749    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0fa8f4-44df-45e7-8028-7d329877afaa-istiod-ca-cert" pod "af0fa8f4-44df-45e7-8028-7d329877afaa" (UID: "af0fa8f4-44df-45e7-8028-7d329877afaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:30.780071 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.780017    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-workload-certs" pod "986e48d0-8cf9-45f3-86c7-06511006a2e1" (UID: "986e48d0-8cf9-45f3-86c7-06511006a2e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.781651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.780199    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-podinfo" pod "af0fa8f4-44df-45e7-8028-7d329877afaa" (UID: "af0fa8f4-44df-45e7-8028-7d329877afaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:30.781651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.780238    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-credential-socket" pod "986e48d0-8cf9-45f3-86c7-06511006a2e1" (UID: "986e48d0-8cf9-45f3-86c7-06511006a2e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.781651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.780280    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-credential-socket" pod "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" (UID: "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.781651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.780373    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-workload-socket" pod "af0fa8f4-44df-45e7-8028-7d329877afaa" (UID: "af0fa8f4-44df-45e7-8028-7d329877afaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.781651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.780509    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-data" pod "986e48d0-8cf9-45f3-86c7-06511006a2e1" (UID: "986e48d0-8cf9-45f3-86c7-06511006a2e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.781651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.780564    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986e48d0-8cf9-45f3-86c7-06511006a2e1-kube-api-access-tsp6l" pod "986e48d0-8cf9-45f3-86c7-06511006a2e1" (UID: "986e48d0-8cf9-45f3-86c7-06511006a2e1"). InnerVolumeSpecName "kube-api-access-tsp6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:30.781651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.780693    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-workload-certs" pod "af0fa8f4-44df-45e7-8028-7d329877afaa" (UID: "af0fa8f4-44df-45e7-8028-7d329877afaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.781651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.780846    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istiod-ca-cert" pod "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" (UID: "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:30.782562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.782525    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-token" pod "986e48d0-8cf9-45f3-86c7-06511006a2e1" (UID: "986e48d0-8cf9-45f3-86c7-06511006a2e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:30.783059 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.783023    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-envoy" pod "986e48d0-8cf9-45f3-86c7-06511006a2e1" (UID: "986e48d0-8cf9-45f3-86c7-06511006a2e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.783158 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.783075    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-podinfo" pod "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" (UID: "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:30.783311 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.783289    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-token" pod "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" (UID: "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:30.783420 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.783402    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-kube-api-access-mq68p" pod "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" (UID: "e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3"). InnerVolumeSpecName "kube-api-access-mq68p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:30.868619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.868588    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz" podUID="39813549-9eb7-4cb7-9f08-4bf983b4cebc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:22:30.880318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880297    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0fa8f4-44df-45e7-8028-7d329877afaa-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.880318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880320    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.880462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880329    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsp6l\" (UniqueName: \"kubernetes.io/projected/986e48d0-8cf9-45f3-86c7-06511006a2e1-kube-api-access-tsp6l\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.880462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880339    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.880462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880350    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.880462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880358    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.880462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880367    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.880462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880375    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.880462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880384    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.880462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880392    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0fa8f4-44df-45e7-8028-7d329877afaa-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.880462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880401    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq68p\" (UniqueName: \"kubernetes.io/projected/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-kube-api-access-mq68p\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.880462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880409    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.880462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880417    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.880462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880425    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.880462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880434    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0fa8f4-44df-45e7-8028-7d329877afaa-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.880462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.880442    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/986e48d0-8cf9-45f3-86c7-06511006a2e1-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.890919 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.890893    6506 generic.go:356] "Generic (PLEG): container finished" podID="e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" containerID="e636f1aec15879049649c5f555ae75b91d9e3b5727cbf2971acd5fa5eb0ad2ae" exitCode=0
Nov 03 16:22:30.891047 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.890946    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"
Nov 03 16:22:30.891047 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.890984    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv" event={"ID":"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3","Type":"ContainerDied","Data":"e636f1aec15879049649c5f555ae75b91d9e3b5727cbf2971acd5fa5eb0ad2ae"}
Nov 03 16:22:30.891165 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.891071    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv" event={"ID":"e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3","Type":"ContainerDied","Data":"ebff8b703d8de18fcf96c9e6a1205a6eb71400598ad916bd0af2a41ee4be5226"}
Nov 03 16:22:30.891165 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.891104    6506 scope.go:122] "RemoveContainer" containerID="e636f1aec15879049649c5f555ae75b91d9e3b5727cbf2971acd5fa5eb0ad2ae"
Nov 03 16:22:30.892514 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.892487    6506 generic.go:356] "Generic (PLEG): container finished" podID="986e48d0-8cf9-45f3-86c7-06511006a2e1" containerID="28e7f6ac50570950bd5ce77ffaada26044e4ce5d4cddea9c89db7d8fc0127025" exitCode=0
Nov 03 16:22:30.892644 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.892542    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"
Nov 03 16:22:30.892644 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.892574    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj" event={"ID":"986e48d0-8cf9-45f3-86c7-06511006a2e1","Type":"ContainerDied","Data":"28e7f6ac50570950bd5ce77ffaada26044e4ce5d4cddea9c89db7d8fc0127025"}
Nov 03 16:22:30.892644 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.892620    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj" event={"ID":"986e48d0-8cf9-45f3-86c7-06511006a2e1","Type":"ContainerDied","Data":"0ef9aae596b7d06fdd9f2e58a602ea44f497ee5c4fee0028c0c21a01800ec166"}
Nov 03 16:22:30.894206 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.894182    6506 generic.go:356] "Generic (PLEG): container finished" podID="af0fa8f4-44df-45e7-8028-7d329877afaa" containerID="036dad6f01f50d4737e26f3850a56715895fa42e162ae6a7b82cd0d405d6809d" exitCode=0
Nov 03 16:22:30.894325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.894222    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j" event={"ID":"af0fa8f4-44df-45e7-8028-7d329877afaa","Type":"ContainerDied","Data":"036dad6f01f50d4737e26f3850a56715895fa42e162ae6a7b82cd0d405d6809d"}
Nov 03 16:22:30.894325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.894236    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"
Nov 03 16:22:30.894325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.894300    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j" event={"ID":"af0fa8f4-44df-45e7-8028-7d329877afaa","Type":"ContainerDied","Data":"82f8e994eaad7c8821bb54be99e34d0899cf63b1a91b85ff0a86035aec42ce47"}
Nov 03 16:22:30.904814 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.904762    6506 scope.go:122] "RemoveContainer" containerID="e636f1aec15879049649c5f555ae75b91d9e3b5727cbf2971acd5fa5eb0ad2ae"
Nov 03 16:22:30.905234 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:30.905100    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e636f1aec15879049649c5f555ae75b91d9e3b5727cbf2971acd5fa5eb0ad2ae\": container with ID starting with e636f1aec15879049649c5f555ae75b91d9e3b5727cbf2971acd5fa5eb0ad2ae not found: ID does not exist" containerID="e636f1aec15879049649c5f555ae75b91d9e3b5727cbf2971acd5fa5eb0ad2ae"
Nov 03 16:22:30.905234 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.905132    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e636f1aec15879049649c5f555ae75b91d9e3b5727cbf2971acd5fa5eb0ad2ae"} err="failed to get container status \"e636f1aec15879049649c5f555ae75b91d9e3b5727cbf2971acd5fa5eb0ad2ae\": rpc error: code = NotFound desc = could not find container \"e636f1aec15879049649c5f555ae75b91d9e3b5727cbf2971acd5fa5eb0ad2ae\": container with ID starting with e636f1aec15879049649c5f555ae75b91d9e3b5727cbf2971acd5fa5eb0ad2ae not found: ID does not exist"
Nov 03 16:22:30.905234 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.905149    6506 scope.go:122] "RemoveContainer" containerID="28e7f6ac50570950bd5ce77ffaada26044e4ce5d4cddea9c89db7d8fc0127025"
Nov 03 16:22:30.915194 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.915150    6506 scope.go:122] "RemoveContainer" containerID="28e7f6ac50570950bd5ce77ffaada26044e4ce5d4cddea9c89db7d8fc0127025"
Nov 03 16:22:30.915530 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:30.915483    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e7f6ac50570950bd5ce77ffaada26044e4ce5d4cddea9c89db7d8fc0127025\": container with ID starting with 28e7f6ac50570950bd5ce77ffaada26044e4ce5d4cddea9c89db7d8fc0127025 not found: ID does not exist" containerID="28e7f6ac50570950bd5ce77ffaada26044e4ce5d4cddea9c89db7d8fc0127025"
Nov 03 16:22:30.915631 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.915554    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e7f6ac50570950bd5ce77ffaada26044e4ce5d4cddea9c89db7d8fc0127025"} err="failed to get container status \"28e7f6ac50570950bd5ce77ffaada26044e4ce5d4cddea9c89db7d8fc0127025\": rpc error: code = NotFound desc = could not find container \"28e7f6ac50570950bd5ce77ffaada26044e4ce5d4cddea9c89db7d8fc0127025\": container with ID starting with 28e7f6ac50570950bd5ce77ffaada26044e4ce5d4cddea9c89db7d8fc0127025 not found: ID does not exist"
Nov 03 16:22:30.915631 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.915572    6506 scope.go:122] "RemoveContainer" containerID="036dad6f01f50d4737e26f3850a56715895fa42e162ae6a7b82cd0d405d6809d"
Nov 03 16:22:30.927099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.927078    6506 scope.go:122] "RemoveContainer" containerID="036dad6f01f50d4737e26f3850a56715895fa42e162ae6a7b82cd0d405d6809d"
Nov 03 16:22:30.927391 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:30.927374    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036dad6f01f50d4737e26f3850a56715895fa42e162ae6a7b82cd0d405d6809d\": container with ID starting with 036dad6f01f50d4737e26f3850a56715895fa42e162ae6a7b82cd0d405d6809d not found: ID does not exist" containerID="036dad6f01f50d4737e26f3850a56715895fa42e162ae6a7b82cd0d405d6809d"
Nov 03 16:22:30.927459 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.927397    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036dad6f01f50d4737e26f3850a56715895fa42e162ae6a7b82cd0d405d6809d"} err="failed to get container status \"036dad6f01f50d4737e26f3850a56715895fa42e162ae6a7b82cd0d405d6809d\": rpc error: code = NotFound desc = could not find container \"036dad6f01f50d4737e26f3850a56715895fa42e162ae6a7b82cd0d405d6809d\": container with ID starting with 036dad6f01f50d4737e26f3850a56715895fa42e162ae6a7b82cd0d405d6809d not found: ID does not exist"
Nov 03 16:22:30.935066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.935038    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"]
Nov 03 16:22:30.941950 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.941926    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-cdnxj"]
Nov 03 16:22:30.956482 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.956458    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"]
Nov 03 16:22:30.960194 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.960170    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-w7ndv"]
Nov 03 16:22:30.972711 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.972689    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"]
Nov 03 16:22:30.983144 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:30.983120    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-q254j"]
Nov 03 16:22:31.034415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:31.034393    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"
Nov 03 16:22:31.185835 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:31.185780    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f093183a-3336-4dce-8945-f907211722f4-kube-api-access-2c42k\" (UniqueName: \"kubernetes.io/projected/f093183a-3336-4dce-8945-f907211722f4-kube-api-access-2c42k\") pod \"f093183a-3336-4dce-8945-f907211722f4\" (UID: \"f093183a-3336-4dce-8945-f907211722f4\") "
Nov 03 16:22:31.186007 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:31.185880    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f093183a-3336-4dce-8945-f907211722f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f093183a-3336-4dce-8945-f907211722f4-maas-api-openshift-ca\") pod \"f093183a-3336-4dce-8945-f907211722f4\" (UID: \"f093183a-3336-4dce-8945-f907211722f4\") "
Nov 03 16:22:31.186007 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:31.185926    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f093183a-3336-4dce-8945-f907211722f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f093183a-3336-4dce-8945-f907211722f4-maas-api-xks-ca\") pod \"f093183a-3336-4dce-8945-f907211722f4\" (UID: \"f093183a-3336-4dce-8945-f907211722f4\") "
Nov 03 16:22:31.186268 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:31.186240    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f093183a-3336-4dce-8945-f907211722f4-maas-api-openshift-ca" pod "f093183a-3336-4dce-8945-f907211722f4" (UID: "f093183a-3336-4dce-8945-f907211722f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:31.188083 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:31.188061    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f093183a-3336-4dce-8945-f907211722f4-maas-api-xks-ca" pod "f093183a-3336-4dce-8945-f907211722f4" (UID: "f093183a-3336-4dce-8945-f907211722f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:31.188174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:31.188154    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f093183a-3336-4dce-8945-f907211722f4-kube-api-access-2c42k" pod "f093183a-3336-4dce-8945-f907211722f4" (UID: "f093183a-3336-4dce-8945-f907211722f4"). InnerVolumeSpecName "kube-api-access-2c42k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:31.286909 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:31.286836    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c42k\" (UniqueName: \"kubernetes.io/projected/f093183a-3336-4dce-8945-f907211722f4-kube-api-access-2c42k\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:31.286909 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:31.286863    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f093183a-3336-4dce-8945-f907211722f4-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:31.286909 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:31.286875    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f093183a-3336-4dce-8945-f907211722f4-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:31.903143 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:31.903111    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"
Nov 03 16:22:31.903662 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:31.903148    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546" event={"ID":"f093183a-3336-4dce-8945-f907211722f4","Type":"ContainerDied","Data":"d1b5d0b8f87033ac24b955f93cf0eff85b683629a71e9460c5713e42e4c00e6e"}
Nov 03 16:22:31.903662 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:31.903206    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1b5d0b8f87033ac24b955f93cf0eff85b683629a71e9460c5713e42e4c00e6e"
Nov 03 16:22:32.102898 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:32.102868    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="986e48d0-8cf9-45f3-86c7-06511006a2e1" path="/var/lib/kubelet/pods/986e48d0-8cf9-45f3-86c7-06511006a2e1/volumes"
Nov 03 16:22:32.103311 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:32.103299    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af0fa8f4-44df-45e7-8028-7d329877afaa" path="/var/lib/kubelet/pods/af0fa8f4-44df-45e7-8028-7d329877afaa/volumes"
Nov 03 16:22:32.103684 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:32.103673    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" path="/var/lib/kubelet/pods/e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3/volumes"
Nov 03 16:22:32.282511 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:32.282470    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2" podUID="f5d53d56-ee8a-4465-8a5a-06e9064ab695" containerName="payload-processing" containerID="cri-o://c294d8fa8288047d98fbfa384ed565250c49758c4b9a7873ea128d30aeaa021b" gracePeriod=25
Nov 03 16:22:32.311324 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:32.311280    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t" podUID="ccadaa1e-d59a-4231-ad4c-d8f46e7f4365" containerName="payload-pre-processing" containerID="cri-o://e1f0b88be24b3429927184dd1f7e14df0d93aa3ba1a9ee036d331725d9fa1cad" gracePeriod=25
Nov 03 16:22:33.713369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:33.713329    6506 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:33.713777 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:33.713386    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2" podUID="5a76c7da-5ade-41c8-a075-430ccf3e7215" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:34.124261 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.124232    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:22:34.163215 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.159363    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:22:34.174121 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.174085    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:22:34.205961 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.205936    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.220233    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17b47389-9b79-402f-bc9b-d404e0b1e75c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b47389-9b79-402f-bc9b-d404e0b1e75c-istiod-ca-cert\") pod \"17b47389-9b79-402f-bc9b-d404e0b1e75c\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.220356    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b9b08d-2d97-4604-890c-10f6305c0108-kube-api-access-qzbvq\" (UniqueName: \"kubernetes.io/projected/20b9b08d-2d97-4604-890c-10f6305c0108-kube-api-access-qzbvq\") pod \"20b9b08d-2d97-4604-890c-10f6305c0108\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.220587    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119e87d8-1b7b-46db-9ac8-289804f75184-kube-api-access-5qdkq\" (UniqueName: \"kubernetes.io/projected/119e87d8-1b7b-46db-9ac8-289804f75184-kube-api-access-5qdkq\") pod \"119e87d8-1b7b-46db-9ac8-289804f75184\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.220605    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b47389-9b79-402f-bc9b-d404e0b1e75c-istiod-ca-cert" pod "17b47389-9b79-402f-bc9b-d404e0b1e75c" (UID: "17b47389-9b79-402f-bc9b-d404e0b1e75c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.220712    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-envoy\") pod \"17b47389-9b79-402f-bc9b-d404e0b1e75c\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.220759    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/119e87d8-1b7b-46db-9ac8-289804f75184-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119e87d8-1b7b-46db-9ac8-289804f75184-istio-podinfo\") pod \"119e87d8-1b7b-46db-9ac8-289804f75184\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.220823    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-podinfo\") pod \"5a76c7da-5ade-41c8-a075-430ccf3e7215\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.220865    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119e87d8-1b7b-46db-9ac8-289804f75184-istio-token\" (UniqueName: \"kubernetes.io/projected/119e87d8-1b7b-46db-9ac8-289804f75184-istio-token\") pod \"119e87d8-1b7b-46db-9ac8-289804f75184\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.220893    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-workload-socket\") pod \"17b47389-9b79-402f-bc9b-d404e0b1e75c\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.220918    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-data\") pod \"17b47389-9b79-402f-bc9b-d404e0b1e75c\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.220945    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-istio-envoy\") pod \"20b9b08d-2d97-4604-890c-10f6305c0108\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.220981    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b9b08d-2d97-4604-890c-10f6305c0108-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9b08d-2d97-4604-890c-10f6305c0108-istiod-ca-cert\") pod \"20b9b08d-2d97-4604-890c-10f6305c0108\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.221035    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-data\") pod \"5a76c7da-5ade-41c8-a075-430ccf3e7215\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.221078    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-workload-certs\") pod \"20b9b08d-2d97-4604-890c-10f6305c0108\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.221111    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-token\" (UniqueName: \"kubernetes.io/projected/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-token\") pod \"5a76c7da-5ade-41c8-a075-430ccf3e7215\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.221151    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a76c7da-5ade-41c8-a075-430ccf3e7215-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a76c7da-5ade-41c8-a075-430ccf3e7215-istiod-ca-cert\") pod \"5a76c7da-5ade-41c8-a075-430ccf3e7215\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.221215    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-workload-certs\") pod \"119e87d8-1b7b-46db-9ac8-289804f75184\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.221247    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-workload-socket\") pod \"119e87d8-1b7b-46db-9ac8-289804f75184\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.221284    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-credential-socket\") pod \"17b47389-9b79-402f-bc9b-d404e0b1e75c\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.221319    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-workload-socket\") pod \"5a76c7da-5ade-41c8-a075-430ccf3e7215\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.221366    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-istio-data\") pod \"20b9b08d-2d97-4604-890c-10f6305c0108\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.221429    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-podinfo\") pod \"17b47389-9b79-402f-bc9b-d404e0b1e75c\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.221469    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-workload-certs\") pod \"17b47389-9b79-402f-bc9b-d404e0b1e75c\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.221502    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b47389-9b79-402f-bc9b-d404e0b1e75c-kube-api-access-b827t\" (UniqueName: \"kubernetes.io/projected/17b47389-9b79-402f-bc9b-d404e0b1e75c-kube-api-access-b827t\") pod \"17b47389-9b79-402f-bc9b-d404e0b1e75c\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.221589    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-envoy\") pod \"5a76c7da-5ade-41c8-a075-430ccf3e7215\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.221637    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b9b08d-2d97-4604-890c-10f6305c0108-istio-token\" (UniqueName: \"kubernetes.io/projected/20b9b08d-2d97-4604-890c-10f6305c0108-istio-token\") pod \"20b9b08d-2d97-4604-890c-10f6305c0108\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") "
Nov 03 16:22:34.221834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.221683    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-credential-socket\") pod \"119e87d8-1b7b-46db-9ac8-289804f75184\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") "
Nov 03 16:22:34.223628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.222299    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b47389-9b79-402f-bc9b-d404e0b1e75c-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.223628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.222520    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-credential-socket" pod "119e87d8-1b7b-46db-9ac8-289804f75184" (UID: "119e87d8-1b7b-46db-9ac8-289804f75184"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.223628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.222677    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-workload-socket" pod "17b47389-9b79-402f-bc9b-d404e0b1e75c" (UID: "17b47389-9b79-402f-bc9b-d404e0b1e75c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.223628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.222862    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-data" pod "17b47389-9b79-402f-bc9b-d404e0b1e75c" (UID: "17b47389-9b79-402f-bc9b-d404e0b1e75c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.226977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.224942    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-credential-socket" pod "17b47389-9b79-402f-bc9b-d404e0b1e75c" (UID: "17b47389-9b79-402f-bc9b-d404e0b1e75c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.226977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.225180    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b9b08d-2d97-4604-890c-10f6305c0108-istiod-ca-cert" pod "20b9b08d-2d97-4604-890c-10f6305c0108" (UID: "20b9b08d-2d97-4604-890c-10f6305c0108"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:34.226977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.225392    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b9b08d-2d97-4604-890c-10f6305c0108-kube-api-access-qzbvq" pod "20b9b08d-2d97-4604-890c-10f6305c0108" (UID: "20b9b08d-2d97-4604-890c-10f6305c0108"). InnerVolumeSpecName "kube-api-access-qzbvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:34.226977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.225463    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a76c7da-5ade-41c8-a075-430ccf3e7215-istiod-ca-cert" pod "5a76c7da-5ade-41c8-a075-430ccf3e7215" (UID: "5a76c7da-5ade-41c8-a075-430ccf3e7215"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:34.226977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.225596    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-workload-certs" pod "17b47389-9b79-402f-bc9b-d404e0b1e75c" (UID: "17b47389-9b79-402f-bc9b-d404e0b1e75c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.226977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.225763    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-workload-socket" pod "5a76c7da-5ade-41c8-a075-430ccf3e7215" (UID: "5a76c7da-5ade-41c8-a075-430ccf3e7215"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.226977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.225943    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-istio-data" pod "20b9b08d-2d97-4604-890c-10f6305c0108" (UID: "20b9b08d-2d97-4604-890c-10f6305c0108"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.226977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.225941    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-workload-certs" pod "119e87d8-1b7b-46db-9ac8-289804f75184" (UID: "119e87d8-1b7b-46db-9ac8-289804f75184"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.226977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.226119    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-workload-socket" pod "119e87d8-1b7b-46db-9ac8-289804f75184" (UID: "119e87d8-1b7b-46db-9ac8-289804f75184"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.226977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.226289    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/119e87d8-1b7b-46db-9ac8-289804f75184-istio-podinfo" pod "119e87d8-1b7b-46db-9ac8-289804f75184" (UID: "119e87d8-1b7b-46db-9ac8-289804f75184"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:34.226977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.226312    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-data" pod "5a76c7da-5ade-41c8-a075-430ccf3e7215" (UID: "5a76c7da-5ade-41c8-a075-430ccf3e7215"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.226977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.226518    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-workload-certs" pod "20b9b08d-2d97-4604-890c-10f6305c0108" (UID: "20b9b08d-2d97-4604-890c-10f6305c0108"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.226977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.226580    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-envoy" pod "17b47389-9b79-402f-bc9b-d404e0b1e75c" (UID: "17b47389-9b79-402f-bc9b-d404e0b1e75c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.226977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.226650    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119e87d8-1b7b-46db-9ac8-289804f75184-istio-token" pod "119e87d8-1b7b-46db-9ac8-289804f75184" (UID: "119e87d8-1b7b-46db-9ac8-289804f75184"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:34.226977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.226702    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119e87d8-1b7b-46db-9ac8-289804f75184-kube-api-access-5qdkq" pod "119e87d8-1b7b-46db-9ac8-289804f75184" (UID: "119e87d8-1b7b-46db-9ac8-289804f75184"). InnerVolumeSpecName "kube-api-access-5qdkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:34.227624 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.227474    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-token" pod "5a76c7da-5ade-41c8-a075-430ccf3e7215" (UID: "5a76c7da-5ade-41c8-a075-430ccf3e7215"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:34.228328 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.228300    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-istio-envoy" pod "20b9b08d-2d97-4604-890c-10f6305c0108" (UID: "20b9b08d-2d97-4604-890c-10f6305c0108"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.228712 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.228690    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-podinfo" pod "17b47389-9b79-402f-bc9b-d404e0b1e75c" (UID: "17b47389-9b79-402f-bc9b-d404e0b1e75c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:34.230011 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.229969    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-podinfo" pod "5a76c7da-5ade-41c8-a075-430ccf3e7215" (UID: "5a76c7da-5ade-41c8-a075-430ccf3e7215"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:34.230102 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.230040    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-envoy" pod "5a76c7da-5ade-41c8-a075-430ccf3e7215" (UID: "5a76c7da-5ade-41c8-a075-430ccf3e7215"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.230102 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.230055    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b9b08d-2d97-4604-890c-10f6305c0108-istio-token" pod "20b9b08d-2d97-4604-890c-10f6305c0108" (UID: "20b9b08d-2d97-4604-890c-10f6305c0108"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:34.231235 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.231214    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b47389-9b79-402f-bc9b-d404e0b1e75c-kube-api-access-b827t" pod "17b47389-9b79-402f-bc9b-d404e0b1e75c" (UID: "17b47389-9b79-402f-bc9b-d404e0b1e75c"). InnerVolumeSpecName "kube-api-access-b827t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:34.322891 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.322858    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-workload-socket\") pod \"20b9b08d-2d97-4604-890c-10f6305c0108\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") "
Nov 03 16:22:34.322891 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.322896    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-istio-envoy\") pod \"119e87d8-1b7b-46db-9ac8-289804f75184\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") "
Nov 03 16:22:34.323132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.322915    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20b9b08d-2d97-4604-890c-10f6305c0108-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9b08d-2d97-4604-890c-10f6305c0108-istio-podinfo\") pod \"20b9b08d-2d97-4604-890c-10f6305c0108\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") "
Nov 03 16:22:34.323132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.322934    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-credential-socket\") pod \"5a76c7da-5ade-41c8-a075-430ccf3e7215\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") "
Nov 03 16:22:34.323132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.322952    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/119e87d8-1b7b-46db-9ac8-289804f75184-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119e87d8-1b7b-46db-9ac8-289804f75184-istiod-ca-cert\") pod \"119e87d8-1b7b-46db-9ac8-289804f75184\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") "
Nov 03 16:22:34.323132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.323077    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-credential-socket\") pod \"20b9b08d-2d97-4604-890c-10f6305c0108\" (UID: \"20b9b08d-2d97-4604-890c-10f6305c0108\") "
Nov 03 16:22:34.323132 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.323121    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-workload-certs\") pod \"5a76c7da-5ade-41c8-a075-430ccf3e7215\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") "
Nov 03 16:22:34.323406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.323159    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-istio-data\") pod \"119e87d8-1b7b-46db-9ac8-289804f75184\" (UID: \"119e87d8-1b7b-46db-9ac8-289804f75184\") "
Nov 03 16:22:34.323406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.323166    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-workload-socket" pod "20b9b08d-2d97-4604-890c-10f6305c0108" (UID: "20b9b08d-2d97-4604-890c-10f6305c0108"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.323406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.323176    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-credential-socket" pod "5a76c7da-5ade-41c8-a075-430ccf3e7215" (UID: "5a76c7da-5ade-41c8-a075-430ccf3e7215"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.323406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.323195    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a76c7da-5ade-41c8-a075-430ccf3e7215-kube-api-access-2z9gd\" (UniqueName: \"kubernetes.io/projected/5a76c7da-5ade-41c8-a075-430ccf3e7215-kube-api-access-2z9gd\") pod \"5a76c7da-5ade-41c8-a075-430ccf3e7215\" (UID: \"5a76c7da-5ade-41c8-a075-430ccf3e7215\") "
Nov 03 16:22:34.323406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.323238    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-token\" (UniqueName: \"kubernetes.io/projected/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-token\") pod \"17b47389-9b79-402f-bc9b-d404e0b1e75c\" (UID: \"17b47389-9b79-402f-bc9b-d404e0b1e75c\") "
Nov 03 16:22:34.323406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.323255    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119e87d8-1b7b-46db-9ac8-289804f75184-istiod-ca-cert" pod "119e87d8-1b7b-46db-9ac8-289804f75184" (UID: "119e87d8-1b7b-46db-9ac8-289804f75184"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:34.323406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.323398    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-credential-socket" pod "20b9b08d-2d97-4604-890c-10f6305c0108" (UID: "20b9b08d-2d97-4604-890c-10f6305c0108"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.323839 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.323411    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-istio-data" pod "119e87d8-1b7b-46db-9ac8-289804f75184" (UID: "119e87d8-1b7b-46db-9ac8-289804f75184"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.323839 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.323418    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-workload-certs" pod "5a76c7da-5ade-41c8-a075-430ccf3e7215" (UID: "5a76c7da-5ade-41c8-a075-430ccf3e7215"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.323839 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.323769    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9b08d-2d97-4604-890c-10f6305c0108-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.323839 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.323827    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324013 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.323844    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324013 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.323923    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324100 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324033    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a76c7da-5ade-41c8-a075-430ccf3e7215-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324100 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324048    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324100 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324064    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324100 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324079    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324100 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324093    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324107    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324121    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324136    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324151    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b827t\" (UniqueName: \"kubernetes.io/projected/17b47389-9b79-402f-bc9b-d404e0b1e75c-kube-api-access-b827t\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324167    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324182    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b9b08d-2d97-4604-890c-10f6305c0108-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324197    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324210    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324226    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324242    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119e87d8-1b7b-46db-9ac8-289804f75184-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324256    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324270    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a76c7da-5ade-41c8-a075-430ccf3e7215-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324284    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324301    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzbvq\" (UniqueName: \"kubernetes.io/projected/20b9b08d-2d97-4604-890c-10f6305c0108-kube-api-access-qzbvq\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324315    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qdkq\" (UniqueName: \"kubernetes.io/projected/119e87d8-1b7b-46db-9ac8-289804f75184-kube-api-access-5qdkq\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324330    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324343    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119e87d8-1b7b-46db-9ac8-289804f75184-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324358    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a76c7da-5ade-41c8-a075-430ccf3e7215-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324373    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119e87d8-1b7b-46db-9ac8-289804f75184-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324388    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324402    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.324534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.324415    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9b08d-2d97-4604-890c-10f6305c0108-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.325241 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.325219    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-istio-envoy" pod "119e87d8-1b7b-46db-9ac8-289804f75184" (UID: "119e87d8-1b7b-46db-9ac8-289804f75184"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.325495 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.325473    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a76c7da-5ade-41c8-a075-430ccf3e7215-kube-api-access-2z9gd" pod "5a76c7da-5ade-41c8-a075-430ccf3e7215" (UID: "5a76c7da-5ade-41c8-a075-430ccf3e7215"). InnerVolumeSpecName "kube-api-access-2z9gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:34.325668 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.325651    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20b9b08d-2d97-4604-890c-10f6305c0108-istio-podinfo" pod "20b9b08d-2d97-4604-890c-10f6305c0108" (UID: "20b9b08d-2d97-4604-890c-10f6305c0108"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:34.325744 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.325728    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-token" pod "17b47389-9b79-402f-bc9b-d404e0b1e75c" (UID: "17b47389-9b79-402f-bc9b-d404e0b1e75c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:34.425431 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.425397    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119e87d8-1b7b-46db-9ac8-289804f75184-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.425431 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.425425    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9b08d-2d97-4604-890c-10f6305c0108-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.425598 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.425438    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z9gd\" (UniqueName: \"kubernetes.io/projected/5a76c7da-5ade-41c8-a075-430ccf3e7215-kube-api-access-2z9gd\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.425598 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.425452    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b47389-9b79-402f-bc9b-d404e0b1e75c-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.930863 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.930828    6506 generic.go:356] "Generic (PLEG): container finished" podID="5a76c7da-5ade-41c8-a075-430ccf3e7215" containerID="90b4c690b6dd52b9577f008d24d2b39d7c076fa2fd75e0a421c73138c6cbbe4c" exitCode=0
Nov 03 16:22:34.931290 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.930871    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"
Nov 03 16:22:34.931290 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.930894    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2" event={"ID":"5a76c7da-5ade-41c8-a075-430ccf3e7215","Type":"ContainerDied","Data":"90b4c690b6dd52b9577f008d24d2b39d7c076fa2fd75e0a421c73138c6cbbe4c"}
Nov 03 16:22:34.931290 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.930943    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2" event={"ID":"5a76c7da-5ade-41c8-a075-430ccf3e7215","Type":"ContainerDied","Data":"2138513c6571110321c2eab24307d0b16b7d9e29b4af2ea2db9f709131533b72"}
Nov 03 16:22:34.931290 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.930963    6506 scope.go:122] "RemoveContainer" containerID="90b4c690b6dd52b9577f008d24d2b39d7c076fa2fd75e0a421c73138c6cbbe4c"
Nov 03 16:22:34.932649 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.932631    6506 generic.go:356] "Generic (PLEG): container finished" podID="17b47389-9b79-402f-bc9b-d404e0b1e75c" containerID="b46da3dbf1aa72f92e8acec9d704c7c5b0d23b8513f3038f0e8e0eadfc51dc57" exitCode=0
Nov 03 16:22:34.932749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.932712    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"
Nov 03 16:22:34.932749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.932715    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556" event={"ID":"17b47389-9b79-402f-bc9b-d404e0b1e75c","Type":"ContainerDied","Data":"b46da3dbf1aa72f92e8acec9d704c7c5b0d23b8513f3038f0e8e0eadfc51dc57"}
Nov 03 16:22:34.932871 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.932765    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556" event={"ID":"17b47389-9b79-402f-bc9b-d404e0b1e75c","Type":"ContainerDied","Data":"3dec0774310ed9c04d15862b7ee95beb8c7ef12dfbecbc35775951ad13b6a413"}
Nov 03 16:22:34.934695 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.934671    6506 generic.go:356] "Generic (PLEG): container finished" podID="20b9b08d-2d97-4604-890c-10f6305c0108" containerID="7234d9df4269a85af5e0ed9fe525aaa89ea6ed11dbe23c25ddeec25105b25298" exitCode=0
Nov 03 16:22:34.934822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.934727    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"
Nov 03 16:22:34.934822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.934789    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw" event={"ID":"20b9b08d-2d97-4604-890c-10f6305c0108","Type":"ContainerDied","Data":"7234d9df4269a85af5e0ed9fe525aaa89ea6ed11dbe23c25ddeec25105b25298"}
Nov 03 16:22:34.934956 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.934845    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw" event={"ID":"20b9b08d-2d97-4604-890c-10f6305c0108","Type":"ContainerDied","Data":"b0f4aae46f31754dd634d4a97c8eccfee23c377326f29624fe61c064a6a77d32"}
Nov 03 16:22:34.936416 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.936395    6506 generic.go:356] "Generic (PLEG): container finished" podID="119e87d8-1b7b-46db-9ac8-289804f75184" containerID="1e606cb3a3438c6c9d3240e6ea43b8da8665f969a53943b64aa7ce083b54d288" exitCode=0
Nov 03 16:22:34.936519 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.936442    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml" event={"ID":"119e87d8-1b7b-46db-9ac8-289804f75184","Type":"ContainerDied","Data":"1e606cb3a3438c6c9d3240e6ea43b8da8665f969a53943b64aa7ce083b54d288"}
Nov 03 16:22:34.936519 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.936448    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"
Nov 03 16:22:34.936519 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.936482    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml" event={"ID":"119e87d8-1b7b-46db-9ac8-289804f75184","Type":"ContainerDied","Data":"4ab06d501223496bae6d05c91fabd1c4a99c77bce816866d19f625ea6d75c2b3"}
Nov 03 16:22:34.944147 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.944127    6506 scope.go:122] "RemoveContainer" containerID="90b4c690b6dd52b9577f008d24d2b39d7c076fa2fd75e0a421c73138c6cbbe4c"
Nov 03 16:22:34.944479 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:34.944455    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b4c690b6dd52b9577f008d24d2b39d7c076fa2fd75e0a421c73138c6cbbe4c\": container with ID starting with 90b4c690b6dd52b9577f008d24d2b39d7c076fa2fd75e0a421c73138c6cbbe4c not found: ID does not exist" containerID="90b4c690b6dd52b9577f008d24d2b39d7c076fa2fd75e0a421c73138c6cbbe4c"
Nov 03 16:22:34.944583 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.944487    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b4c690b6dd52b9577f008d24d2b39d7c076fa2fd75e0a421c73138c6cbbe4c"} err="failed to get container status \"90b4c690b6dd52b9577f008d24d2b39d7c076fa2fd75e0a421c73138c6cbbe4c\": rpc error: code = NotFound desc = could not find container \"90b4c690b6dd52b9577f008d24d2b39d7c076fa2fd75e0a421c73138c6cbbe4c\": container with ID starting with 90b4c690b6dd52b9577f008d24d2b39d7c076fa2fd75e0a421c73138c6cbbe4c not found: ID does not exist"
Nov 03 16:22:34.944583 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.944509    6506 scope.go:122] "RemoveContainer" containerID="b46da3dbf1aa72f92e8acec9d704c7c5b0d23b8513f3038f0e8e0eadfc51dc57"
Nov 03 16:22:34.954903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.954731    6506 scope.go:122] "RemoveContainer" containerID="b46da3dbf1aa72f92e8acec9d704c7c5b0d23b8513f3038f0e8e0eadfc51dc57"
Nov 03 16:22:34.955046 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:34.955028    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b46da3dbf1aa72f92e8acec9d704c7c5b0d23b8513f3038f0e8e0eadfc51dc57\": container with ID starting with b46da3dbf1aa72f92e8acec9d704c7c5b0d23b8513f3038f0e8e0eadfc51dc57 not found: ID does not exist" containerID="b46da3dbf1aa72f92e8acec9d704c7c5b0d23b8513f3038f0e8e0eadfc51dc57"
Nov 03 16:22:34.955111 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.955052    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46da3dbf1aa72f92e8acec9d704c7c5b0d23b8513f3038f0e8e0eadfc51dc57"} err="failed to get container status \"b46da3dbf1aa72f92e8acec9d704c7c5b0d23b8513f3038f0e8e0eadfc51dc57\": rpc error: code = NotFound desc = could not find container \"b46da3dbf1aa72f92e8acec9d704c7c5b0d23b8513f3038f0e8e0eadfc51dc57\": container with ID starting with b46da3dbf1aa72f92e8acec9d704c7c5b0d23b8513f3038f0e8e0eadfc51dc57 not found: ID does not exist"
Nov 03 16:22:34.955111 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.955068    6506 scope.go:122] "RemoveContainer" containerID="7234d9df4269a85af5e0ed9fe525aaa89ea6ed11dbe23c25ddeec25105b25298"
Nov 03 16:22:34.964679 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.964448    6506 scope.go:122] "RemoveContainer" containerID="1551a6edd5f3fcca5c624548874ffe5096f2c67fe1cd90d38c89ec5e8d7f553a"
Nov 03 16:22:34.968378 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.968355    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"]
Nov 03 16:22:34.972514 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.972475    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-hcnp2"]
Nov 03 16:22:34.974650 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.974637    6506 scope.go:122] "RemoveContainer" containerID="7234d9df4269a85af5e0ed9fe525aaa89ea6ed11dbe23c25ddeec25105b25298"
Nov 03 16:22:34.974932 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:34.974916    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7234d9df4269a85af5e0ed9fe525aaa89ea6ed11dbe23c25ddeec25105b25298\": container with ID starting with 7234d9df4269a85af5e0ed9fe525aaa89ea6ed11dbe23c25ddeec25105b25298 not found: ID does not exist" containerID="7234d9df4269a85af5e0ed9fe525aaa89ea6ed11dbe23c25ddeec25105b25298"
Nov 03 16:22:34.974990 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.974939    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7234d9df4269a85af5e0ed9fe525aaa89ea6ed11dbe23c25ddeec25105b25298"} err="failed to get container status \"7234d9df4269a85af5e0ed9fe525aaa89ea6ed11dbe23c25ddeec25105b25298\": rpc error: code = NotFound desc = could not find container \"7234d9df4269a85af5e0ed9fe525aaa89ea6ed11dbe23c25ddeec25105b25298\": container with ID starting with 7234d9df4269a85af5e0ed9fe525aaa89ea6ed11dbe23c25ddeec25105b25298 not found: ID does not exist"
Nov 03 16:22:34.974990 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.974956    6506 scope.go:122] "RemoveContainer" containerID="1551a6edd5f3fcca5c624548874ffe5096f2c67fe1cd90d38c89ec5e8d7f553a"
Nov 03 16:22:34.975217 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:34.975197    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1551a6edd5f3fcca5c624548874ffe5096f2c67fe1cd90d38c89ec5e8d7f553a\": container with ID starting with 1551a6edd5f3fcca5c624548874ffe5096f2c67fe1cd90d38c89ec5e8d7f553a not found: ID does not exist" containerID="1551a6edd5f3fcca5c624548874ffe5096f2c67fe1cd90d38c89ec5e8d7f553a"
Nov 03 16:22:34.975249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.975225    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1551a6edd5f3fcca5c624548874ffe5096f2c67fe1cd90d38c89ec5e8d7f553a"} err="failed to get container status \"1551a6edd5f3fcca5c624548874ffe5096f2c67fe1cd90d38c89ec5e8d7f553a\": rpc error: code = NotFound desc = could not find container \"1551a6edd5f3fcca5c624548874ffe5096f2c67fe1cd90d38c89ec5e8d7f553a\": container with ID starting with 1551a6edd5f3fcca5c624548874ffe5096f2c67fe1cd90d38c89ec5e8d7f553a not found: ID does not exist"
Nov 03 16:22:34.975249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.975240    6506 scope.go:122] "RemoveContainer" containerID="1e606cb3a3438c6c9d3240e6ea43b8da8665f969a53943b64aa7ce083b54d288"
Nov 03 16:22:34.984626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.984608    6506 scope.go:122] "RemoveContainer" containerID="1e606cb3a3438c6c9d3240e6ea43b8da8665f969a53943b64aa7ce083b54d288"
Nov 03 16:22:34.984913 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:34.984896    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e606cb3a3438c6c9d3240e6ea43b8da8665f969a53943b64aa7ce083b54d288\": container with ID starting with 1e606cb3a3438c6c9d3240e6ea43b8da8665f969a53943b64aa7ce083b54d288 not found: ID does not exist" containerID="1e606cb3a3438c6c9d3240e6ea43b8da8665f969a53943b64aa7ce083b54d288"
Nov 03 16:22:34.984971 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.984919    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e606cb3a3438c6c9d3240e6ea43b8da8665f969a53943b64aa7ce083b54d288"} err="failed to get container status \"1e606cb3a3438c6c9d3240e6ea43b8da8665f969a53943b64aa7ce083b54d288\": rpc error: code = NotFound desc = could not find container \"1e606cb3a3438c6c9d3240e6ea43b8da8665f969a53943b64aa7ce083b54d288\": container with ID starting with 1e606cb3a3438c6c9d3240e6ea43b8da8665f969a53943b64aa7ce083b54d288 not found: ID does not exist"
Nov 03 16:22:34.985510 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.985492    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"]
Nov 03 16:22:34.990142 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:34.990123    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-5t556"]
Nov 03 16:22:35.003028 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:35.003003    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"]
Nov 03 16:22:35.007486 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:35.007465    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ldvml"]
Nov 03 16:22:35.019966 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:35.019944    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"]
Nov 03 16:22:35.024083 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:35.024065    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-9jmgw"]
Nov 03 16:22:36.104673 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:36.104638    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="119e87d8-1b7b-46db-9ac8-289804f75184" path="/var/lib/kubelet/pods/119e87d8-1b7b-46db-9ac8-289804f75184/volumes"
Nov 03 16:22:36.105342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:36.105317    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17b47389-9b79-402f-bc9b-d404e0b1e75c" path="/var/lib/kubelet/pods/17b47389-9b79-402f-bc9b-d404e0b1e75c/volumes"
Nov 03 16:22:36.105850 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:36.105831    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b9b08d-2d97-4604-890c-10f6305c0108" path="/var/lib/kubelet/pods/20b9b08d-2d97-4604-890c-10f6305c0108/volumes"
Nov 03 16:22:36.106630 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:36.106609    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a76c7da-5ade-41c8-a075-430ccf3e7215" path="/var/lib/kubelet/pods/5a76c7da-5ade-41c8-a075-430ccf3e7215/volumes"
Nov 03 16:22:36.720823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:36.717135    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:36.839595 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:36.839550    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main" containerID="cri-o://8ccdd1537bfd5d5df8316ed0a27f604e97e1abecad3ea3284341e8e551a778e2" gracePeriod=45
Nov 03 16:22:36.959097 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:36.959070    6506 generic.go:356] "Generic (PLEG): container finished" podID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerID="8ccdd1537bfd5d5df8316ed0a27f604e97e1abecad3ea3284341e8e551a778e2" exitCode=0
Nov 03 16:22:36.959177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:36.959152    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" event={"ID":"c7fb2203-adc1-4dcb-b092-42f6d5296165","Type":"ContainerDied","Data":"8ccdd1537bfd5d5df8316ed0a27f604e97e1abecad3ea3284341e8e551a778e2"}
Nov 03 16:22:36.961606 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:36.961577    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:37.108604 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.106852    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"]
Nov 03 16:22:37.108604 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.107109    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc" podUID="27399426-17f7-4ea7-aba6-12338b82c6f3" containerName="istio-proxy" containerID="cri-o://6af2f3ffed58b75a271a821521541f45e11b196e361df4762fd1995408c88343" gracePeriod=30
Nov 03 16:22:37.127013 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.123887    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"]
Nov 03 16:22:37.127013 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.124151    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy" containerID="cri-o://94c0489fc4c6f194c5baf122536b4b4d41c49209483c84bdcb86b15aa4959887" gracePeriod=30
Nov 03 16:22:37.128830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.127962    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"]
Nov 03 16:22:37.128830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.128162    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894" podUID="de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" containerName="istio-proxy" containerID="cri-o://a6c706849899f6371030aa95b8e314fe6d9094638ed61d2543d8785697757d45" gracePeriod=30
Nov 03 16:22:37.178837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.178808    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:22:37.260907 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.260863    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:37.304776 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.304745    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-dshm\") pod \"c7fb2203-adc1-4dcb-b092-42f6d5296165\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") "
Nov 03 16:22:37.304942 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.304865    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7fb2203-adc1-4dcb-b092-42f6d5296165-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fb2203-adc1-4dcb-b092-42f6d5296165-tls-certs\") pod \"c7fb2203-adc1-4dcb-b092-42f6d5296165\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") "
Nov 03 16:22:37.304942 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.304916    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7fb2203-adc1-4dcb-b092-42f6d5296165-kube-api-access-dmmh6\" (UniqueName: \"kubernetes.io/projected/c7fb2203-adc1-4dcb-b092-42f6d5296165-kube-api-access-dmmh6\") pod \"c7fb2203-adc1-4dcb-b092-42f6d5296165\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") "
Nov 03 16:22:37.305070 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.304971    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-model-cache\") pod \"c7fb2203-adc1-4dcb-b092-42f6d5296165\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") "
Nov 03 16:22:37.305070 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.305023    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-tmp-dir\") pod \"c7fb2203-adc1-4dcb-b092-42f6d5296165\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") "
Nov 03 16:22:37.305174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.305079    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-home\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-home\") pod \"c7fb2203-adc1-4dcb-b092-42f6d5296165\" (UID: \"c7fb2203-adc1-4dcb-b092-42f6d5296165\") "
Nov 03 16:22:37.306144 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.305736    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-home" pod "c7fb2203-adc1-4dcb-b092-42f6d5296165" (UID: "c7fb2203-adc1-4dcb-b092-42f6d5296165"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:37.306144 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.305982    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-model-cache" pod "c7fb2203-adc1-4dcb-b092-42f6d5296165" (UID: "c7fb2203-adc1-4dcb-b092-42f6d5296165"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:37.306414 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.306366    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-tmp-dir" pod "c7fb2203-adc1-4dcb-b092-42f6d5296165" (UID: "c7fb2203-adc1-4dcb-b092-42f6d5296165"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:37.309313 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.309278    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fb2203-adc1-4dcb-b092-42f6d5296165-tls-certs" pod "c7fb2203-adc1-4dcb-b092-42f6d5296165" (UID: "c7fb2203-adc1-4dcb-b092-42f6d5296165"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:37.311361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.311314    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fb2203-adc1-4dcb-b092-42f6d5296165-kube-api-access-dmmh6" pod "c7fb2203-adc1-4dcb-b092-42f6d5296165" (UID: "c7fb2203-adc1-4dcb-b092-42f6d5296165"). InnerVolumeSpecName "kube-api-access-dmmh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:37.313538 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.313512    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-dshm" pod "c7fb2203-adc1-4dcb-b092-42f6d5296165" (UID: "c7fb2203-adc1-4dcb-b092-42f6d5296165"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:37.407080 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.407045    6506 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fb2203-adc1-4dcb-b092-42f6d5296165-tls-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:37.407080 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.407084    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmmh6\" (UniqueName: \"kubernetes.io/projected/c7fb2203-adc1-4dcb-b092-42f6d5296165-kube-api-access-dmmh6\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:37.407325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.407105    6506 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-model-cache\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:37.407325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.407119    6506 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-tmp-dir\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:37.407325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.407135    6506 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-home\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:37.407325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.407146    6506 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fb2203-adc1-4dcb-b092-42f6d5296165-dshm\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:37.442492 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.442462    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"
Nov 03 16:22:37.509401 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.509361    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39813549-9eb7-4cb7-9f08-4bf983b4cebc-kube-api-access-4242c\" (UniqueName: \"kubernetes.io/projected/39813549-9eb7-4cb7-9f08-4bf983b4cebc-kube-api-access-4242c\") pod \"39813549-9eb7-4cb7-9f08-4bf983b4cebc\" (UID: \"39813549-9eb7-4cb7-9f08-4bf983b4cebc\") "
Nov 03 16:22:37.509639 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.509616    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39813549-9eb7-4cb7-9f08-4bf983b4cebc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39813549-9eb7-4cb7-9f08-4bf983b4cebc-plugins-config-volume\") pod \"39813549-9eb7-4cb7-9f08-4bf983b4cebc\" (UID: \"39813549-9eb7-4cb7-9f08-4bf983b4cebc\") "
Nov 03 16:22:37.510339 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.510309    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39813549-9eb7-4cb7-9f08-4bf983b4cebc-plugins-config-volume" pod "39813549-9eb7-4cb7-9f08-4bf983b4cebc" (UID: "39813549-9eb7-4cb7-9f08-4bf983b4cebc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:37.512701 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.512679    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39813549-9eb7-4cb7-9f08-4bf983b4cebc-kube-api-access-4242c" pod "39813549-9eb7-4cb7-9f08-4bf983b4cebc" (UID: "39813549-9eb7-4cb7-9f08-4bf983b4cebc"). InnerVolumeSpecName "kube-api-access-4242c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:37.611581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.611506    6506 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39813549-9eb7-4cb7-9f08-4bf983b4cebc-plugins-config-volume\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:37.611581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.611543    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4242c\" (UniqueName: \"kubernetes.io/projected/39813549-9eb7-4cb7-9f08-4bf983b4cebc-kube-api-access-4242c\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:37.970714 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.970636    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"
Nov 03 16:22:37.970714 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.970652    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g" event={"ID":"c7fb2203-adc1-4dcb-b092-42f6d5296165","Type":"ContainerDied","Data":"06bb32be6465a62202a4806a079f21d46884004870468762c492832a4b01379e"}
Nov 03 16:22:37.970967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.970715    6506 scope.go:122] "RemoveContainer" containerID="8ccdd1537bfd5d5df8316ed0a27f604e97e1abecad3ea3284341e8e551a778e2"
Nov 03 16:22:37.972228 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.972196    6506 generic.go:356] "Generic (PLEG): container finished" podID="39813549-9eb7-4cb7-9f08-4bf983b4cebc" containerID="119250b5231ab824d8f19f7131df5e0ceb47586ebb4089e30acd9c790c9f6a29" exitCode=0
Nov 03 16:22:37.972338 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.972282    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz" event={"ID":"39813549-9eb7-4cb7-9f08-4bf983b4cebc","Type":"ContainerDied","Data":"119250b5231ab824d8f19f7131df5e0ceb47586ebb4089e30acd9c790c9f6a29"}
Nov 03 16:22:37.972338 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.972331    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz" event={"ID":"39813549-9eb7-4cb7-9f08-4bf983b4cebc","Type":"ContainerDied","Data":"cb6ddd490e72c4bb96f7a3634a095526b434afbaea241b699406c6699ed38a3e"}
Nov 03 16:22:37.972458 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.972404    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"
Nov 03 16:22:37.983301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.983228    6506 scope.go:122] "RemoveContainer" containerID="119250b5231ab824d8f19f7131df5e0ceb47586ebb4089e30acd9c790c9f6a29"
Nov 03 16:22:37.993650 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.993632    6506 scope.go:122] "RemoveContainer" containerID="119250b5231ab824d8f19f7131df5e0ceb47586ebb4089e30acd9c790c9f6a29"
Nov 03 16:22:37.993935 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:37.993911    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119250b5231ab824d8f19f7131df5e0ceb47586ebb4089e30acd9c790c9f6a29\": container with ID starting with 119250b5231ab824d8f19f7131df5e0ceb47586ebb4089e30acd9c790c9f6a29 not found: ID does not exist" containerID="119250b5231ab824d8f19f7131df5e0ceb47586ebb4089e30acd9c790c9f6a29"
Nov 03 16:22:37.994011 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:37.993948    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119250b5231ab824d8f19f7131df5e0ceb47586ebb4089e30acd9c790c9f6a29"} err="failed to get container status \"119250b5231ab824d8f19f7131df5e0ceb47586ebb4089e30acd9c790c9f6a29\": rpc error: code = NotFound desc = could not find container \"119250b5231ab824d8f19f7131df5e0ceb47586ebb4089e30acd9c790c9f6a29\": container with ID starting with 119250b5231ab824d8f19f7131df5e0ceb47586ebb4089e30acd9c790c9f6a29 not found: ID does not exist"
Nov 03 16:22:38.000374 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:38.000350    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"]
Nov 03 16:22:38.006047 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:38.006024    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/premium-w2-b132e3-kserve-7c7455fb84-zd88g"]
Nov 03 16:22:38.017596 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:38.017575    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"]
Nov 03 16:22:38.022236 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:38.022208    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4a19cd-6c949697c5-xm2hz"]
Nov 03 16:22:38.104860 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:38.104827    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39813549-9eb7-4cb7-9f08-4bf983b4cebc" path="/var/lib/kubelet/pods/39813549-9eb7-4cb7-9f08-4bf983b4cebc/volumes"
Nov 03 16:22:38.105259 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:38.105246    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" path="/var/lib/kubelet/pods/c7fb2203-adc1-4dcb-b092-42f6d5296165/volumes"
Nov 03 16:22:38.888241 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:38.888206    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:40.035700 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:40.035667    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:40.227995 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:40.227959    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:40.253152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:40.253109    6506 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:40.253415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:40.253381    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:40.481341 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:40.481259    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:40.862736 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:40.862699    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:40.867681 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:40.867648    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t" podUID="ccadaa1e-d59a-4231-ad4c-d8f46e7f4365" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:22:40.867681 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:40.867671    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2" podUID="f5d53d56-ee8a-4465-8a5a-06e9064ab695" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:22:42.445099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.445066    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:22:42.472725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.472693    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:22:42.475625 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.475570    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:22:42.579105 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579074    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-credential-socket\") pod \"b24bd837-1361-49af-bb52-dd51f01967a7\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") "
Nov 03 16:22:42.579294 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579115    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27399426-17f7-4ea7-aba6-12338b82c6f3-kube-api-access-wf9jp\" (UniqueName: \"kubernetes.io/projected/27399426-17f7-4ea7-aba6-12338b82c6f3-kube-api-access-wf9jp\") pod \"27399426-17f7-4ea7-aba6-12338b82c6f3\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") "
Nov 03 16:22:42.579294 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579137    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-data\") pod \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") "
Nov 03 16:22:42.579294 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579162    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-istio-data\") pod \"b24bd837-1361-49af-bb52-dd51f01967a7\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") "
Nov 03 16:22:42.579463 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579319    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-data\") pod \"27399426-17f7-4ea7-aba6-12338b82c6f3\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") "
Nov 03 16:22:42.579463 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579381    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24bd837-1361-49af-bb52-dd51f01967a7-istio-token\" (UniqueName: \"kubernetes.io/projected/b24bd837-1361-49af-bb52-dd51f01967a7-istio-token\") pod \"b24bd837-1361-49af-bb52-dd51f01967a7\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") "
Nov 03 16:22:42.579463 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579412    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24bd837-1361-49af-bb52-dd51f01967a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24bd837-1361-49af-bb52-dd51f01967a7-istiod-ca-cert\") pod \"b24bd837-1361-49af-bb52-dd51f01967a7\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") "
Nov 03 16:22:42.579463 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579419    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-credential-socket" pod "b24bd837-1361-49af-bb52-dd51f01967a7" (UID: "b24bd837-1361-49af-bb52-dd51f01967a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.579463 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579449    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-token\" (UniqueName: \"kubernetes.io/projected/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-token\") pod \"27399426-17f7-4ea7-aba6-12338b82c6f3\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") "
Nov 03 16:22:42.579463 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579443    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-data" pod "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" (UID: "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.579856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579491    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27399426-17f7-4ea7-aba6-12338b82c6f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27399426-17f7-4ea7-aba6-12338b82c6f3-istiod-ca-cert\") pod \"27399426-17f7-4ea7-aba6-12338b82c6f3\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") "
Nov 03 16:22:42.579856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579535    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-envoy\") pod \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") "
Nov 03 16:22:42.579856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579574    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-podinfo\") pod \"27399426-17f7-4ea7-aba6-12338b82c6f3\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") "
Nov 03 16:22:42.579856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579582    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-data" pod "27399426-17f7-4ea7-aba6-12338b82c6f3" (UID: "27399426-17f7-4ea7-aba6-12338b82c6f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.579856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579603    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-credential-socket\") pod \"27399426-17f7-4ea7-aba6-12338b82c6f3\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") "
Nov 03 16:22:42.579856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579630    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24bd837-1361-49af-bb52-dd51f01967a7-kube-api-access-mb6sw\" (UniqueName: \"kubernetes.io/projected/b24bd837-1361-49af-bb52-dd51f01967a7-kube-api-access-mb6sw\") pod \"b24bd837-1361-49af-bb52-dd51f01967a7\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") "
Nov 03 16:22:42.579856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579664    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-workload-certs\") pod \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") "
Nov 03 16:22:42.579856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579690    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-podinfo\") pod \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") "
Nov 03 16:22:42.579856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579717    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-kube-api-access-4dk55\" (UniqueName: \"kubernetes.io/projected/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-kube-api-access-4dk55\") pod \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") "
Nov 03 16:22:42.579856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579773    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-workload-socket\") pod \"b24bd837-1361-49af-bb52-dd51f01967a7\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") "
Nov 03 16:22:42.579856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579827    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istiod-ca-cert\") pod \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") "
Nov 03 16:22:42.580427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579878    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-credential-socket\") pod \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") "
Nov 03 16:22:42.580427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579911    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-workload-certs\") pod \"27399426-17f7-4ea7-aba6-12338b82c6f3\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") "
Nov 03 16:22:42.580427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579929    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-istio-data" pod "b24bd837-1361-49af-bb52-dd51f01967a7" (UID: "b24bd837-1361-49af-bb52-dd51f01967a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.580427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579945    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-workload-socket\") pod \"27399426-17f7-4ea7-aba6-12338b82c6f3\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") "
Nov 03 16:22:42.580427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.579942    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27399426-17f7-4ea7-aba6-12338b82c6f3-istiod-ca-cert" pod "27399426-17f7-4ea7-aba6-12338b82c6f3" (UID: "27399426-17f7-4ea7-aba6-12338b82c6f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:42.580427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.580049    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b24bd837-1361-49af-bb52-dd51f01967a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24bd837-1361-49af-bb52-dd51f01967a7-istio-podinfo\") pod \"b24bd837-1361-49af-bb52-dd51f01967a7\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") "
Nov 03 16:22:42.580427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.580086    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-workload-socket\") pod \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") "
Nov 03 16:22:42.580427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.580127    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-istio-envoy\") pod \"b24bd837-1361-49af-bb52-dd51f01967a7\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") "
Nov 03 16:22:42.580427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.580168    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-workload-socket" pod "27399426-17f7-4ea7-aba6-12338b82c6f3" (UID: "27399426-17f7-4ea7-aba6-12338b82c6f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.580427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.580174    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-token\" (UniqueName: \"kubernetes.io/projected/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-token\") pod \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\" (UID: \"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1\") "
Nov 03 16:22:42.580427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.580239    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-envoy\") pod \"27399426-17f7-4ea7-aba6-12338b82c6f3\" (UID: \"27399426-17f7-4ea7-aba6-12338b82c6f3\") "
Nov 03 16:22:42.580427 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.580319    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-workload-certs\") pod \"b24bd837-1361-49af-bb52-dd51f01967a7\" (UID: \"b24bd837-1361-49af-bb52-dd51f01967a7\") "
Nov 03 16:22:42.581101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.580668    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24bd837-1361-49af-bb52-dd51f01967a7-istiod-ca-cert" pod "b24bd837-1361-49af-bb52-dd51f01967a7" (UID: "b24bd837-1361-49af-bb52-dd51f01967a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:42.581101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.580870    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istiod-ca-cert" pod "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" (UID: "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:42.581101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.580966    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.581101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.580987    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.581101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.581002    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.581101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.581017    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.581101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.581031    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24bd837-1361-49af-bb52-dd51f01967a7-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.581101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.581044    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27399426-17f7-4ea7-aba6-12338b82c6f3-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.581101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.581057    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.581101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.581070    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.581101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.581083    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-workload-socket" pod "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" (UID: "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.581667 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.581210    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-workload-certs" pod "27399426-17f7-4ea7-aba6-12338b82c6f3" (UID: "27399426-17f7-4ea7-aba6-12338b82c6f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.581667 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.581256    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-credential-socket" pod "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" (UID: "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.581667 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.581263    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-workload-certs" pod "b24bd837-1361-49af-bb52-dd51f01967a7" (UID: "b24bd837-1361-49af-bb52-dd51f01967a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.581667 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.581475    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-credential-socket" pod "27399426-17f7-4ea7-aba6-12338b82c6f3" (UID: "27399426-17f7-4ea7-aba6-12338b82c6f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.581889 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.581816    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-workload-socket" pod "b24bd837-1361-49af-bb52-dd51f01967a7" (UID: "b24bd837-1361-49af-bb52-dd51f01967a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.582005 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.581978    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-workload-certs" pod "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" (UID: "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.585877 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.585829    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-podinfo" pod "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" (UID: "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:42.585877 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.585868    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24bd837-1361-49af-bb52-dd51f01967a7-istio-token" pod "b24bd837-1361-49af-bb52-dd51f01967a7" (UID: "b24bd837-1361-49af-bb52-dd51f01967a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:42.587312 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.587285    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-token" pod "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" (UID: "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:42.587456 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.587434    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-istio-envoy" pod "b24bd837-1361-49af-bb52-dd51f01967a7" (UID: "b24bd837-1361-49af-bb52-dd51f01967a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.587570 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.587545    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-podinfo" pod "27399426-17f7-4ea7-aba6-12338b82c6f3" (UID: "27399426-17f7-4ea7-aba6-12338b82c6f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:42.587660 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.587602    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24bd837-1361-49af-bb52-dd51f01967a7-kube-api-access-mb6sw" pod "b24bd837-1361-49af-bb52-dd51f01967a7" (UID: "b24bd837-1361-49af-bb52-dd51f01967a7"). InnerVolumeSpecName "kube-api-access-mb6sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:42.588201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.588086    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b24bd837-1361-49af-bb52-dd51f01967a7-istio-podinfo" pod "b24bd837-1361-49af-bb52-dd51f01967a7" (UID: "b24bd837-1361-49af-bb52-dd51f01967a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:42.588201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.588108    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27399426-17f7-4ea7-aba6-12338b82c6f3-kube-api-access-wf9jp" pod "27399426-17f7-4ea7-aba6-12338b82c6f3" (UID: "27399426-17f7-4ea7-aba6-12338b82c6f3"). InnerVolumeSpecName "kube-api-access-wf9jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:42.588201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.588185    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-token" pod "27399426-17f7-4ea7-aba6-12338b82c6f3" (UID: "27399426-17f7-4ea7-aba6-12338b82c6f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:42.588339 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.588196    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-envoy" pod "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" (UID: "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.588339 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.588266    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-kube-api-access-4dk55" pod "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" (UID: "de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1"). InnerVolumeSpecName "kube-api-access-4dk55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:42.589417 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.589394    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-envoy" pod "27399426-17f7-4ea7-aba6-12338b82c6f3" (UID: "27399426-17f7-4ea7-aba6-12338b82c6f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.682561 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682489    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682561 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682517    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682561 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682527    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dk55\" (UniqueName: \"kubernetes.io/projected/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-kube-api-access-4dk55\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682561 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682537    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682561 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682546    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682561 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682556    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682561 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682564    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24bd837-1361-49af-bb52-dd51f01967a7-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682572    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682580    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682588    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682595    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682604    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24bd837-1361-49af-bb52-dd51f01967a7-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682614    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf9jp\" (UniqueName: \"kubernetes.io/projected/27399426-17f7-4ea7-aba6-12338b82c6f3-kube-api-access-wf9jp\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682622    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24bd837-1361-49af-bb52-dd51f01967a7-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682629    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682636    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682645    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27399426-17f7-4ea7-aba6-12338b82c6f3-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682652    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27399426-17f7-4ea7-aba6-12338b82c6f3-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.682926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:42.682661    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb6sw\" (UniqueName: \"kubernetes.io/projected/b24bd837-1361-49af-bb52-dd51f01967a7-kube-api-access-mb6sw\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:43.010719 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.010634    6506 generic.go:356] "Generic (PLEG): container finished" podID="de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" containerID="a6c706849899f6371030aa95b8e314fe6d9094638ed61d2543d8785697757d45" exitCode=0
Nov 03 16:22:43.010719 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.010697    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"
Nov 03 16:22:43.010976 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.010716    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894" event={"ID":"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1","Type":"ContainerDied","Data":"a6c706849899f6371030aa95b8e314fe6d9094638ed61d2543d8785697757d45"}
Nov 03 16:22:43.010976 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.010827    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894" event={"ID":"de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1","Type":"ContainerDied","Data":"e2db75b9b1bee08b228cdabde8b969e6c0c9cf997c74660dc9bd458810c79c6c"}
Nov 03 16:22:43.010976 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.010859    6506 scope.go:122] "RemoveContainer" containerID="a6c706849899f6371030aa95b8e314fe6d9094638ed61d2543d8785697757d45"
Nov 03 16:22:43.012345 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.012320    6506 generic.go:356] "Generic (PLEG): container finished" podID="b24bd837-1361-49af-bb52-dd51f01967a7" containerID="94c0489fc4c6f194c5baf122536b4b4d41c49209483c84bdcb86b15aa4959887" exitCode=0
Nov 03 16:22:43.012479 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.012378    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"
Nov 03 16:22:43.012479 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.012430    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" event={"ID":"b24bd837-1361-49af-bb52-dd51f01967a7","Type":"ContainerDied","Data":"94c0489fc4c6f194c5baf122536b4b4d41c49209483c84bdcb86b15aa4959887"}
Nov 03 16:22:43.012608 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.012521    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq" event={"ID":"b24bd837-1361-49af-bb52-dd51f01967a7","Type":"ContainerDied","Data":"34f0365d4a741227a671dab7d49c42ad6af4e9793e0b36619f9d38e36a61c951"}
Nov 03 16:22:43.014114 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.014090    6506 generic.go:356] "Generic (PLEG): container finished" podID="27399426-17f7-4ea7-aba6-12338b82c6f3" containerID="6af2f3ffed58b75a271a821521541f45e11b196e361df4762fd1995408c88343" exitCode=0
Nov 03 16:22:43.014238 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.014145    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"
Nov 03 16:22:43.014238 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.014201    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc" event={"ID":"27399426-17f7-4ea7-aba6-12338b82c6f3","Type":"ContainerDied","Data":"6af2f3ffed58b75a271a821521541f45e11b196e361df4762fd1995408c88343"}
Nov 03 16:22:43.014350 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.014251    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc" event={"ID":"27399426-17f7-4ea7-aba6-12338b82c6f3","Type":"ContainerDied","Data":"fe7ea970654d1da6ea4029883c28ec6f52dcabc9aaa935be9906bd4cc9cf78d8"}
Nov 03 16:22:43.022023 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.022007    6506 scope.go:122] "RemoveContainer" containerID="a6c706849899f6371030aa95b8e314fe6d9094638ed61d2543d8785697757d45"
Nov 03 16:22:43.022401 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:43.022350    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c706849899f6371030aa95b8e314fe6d9094638ed61d2543d8785697757d45\": container with ID starting with a6c706849899f6371030aa95b8e314fe6d9094638ed61d2543d8785697757d45 not found: ID does not exist" containerID="a6c706849899f6371030aa95b8e314fe6d9094638ed61d2543d8785697757d45"
Nov 03 16:22:43.022401 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.022382    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c706849899f6371030aa95b8e314fe6d9094638ed61d2543d8785697757d45"} err="failed to get container status \"a6c706849899f6371030aa95b8e314fe6d9094638ed61d2543d8785697757d45\": rpc error: code = NotFound desc = could not find container \"a6c706849899f6371030aa95b8e314fe6d9094638ed61d2543d8785697757d45\": container with ID starting with a6c706849899f6371030aa95b8e314fe6d9094638ed61d2543d8785697757d45 not found: ID does not exist"
Nov 03 16:22:43.022526 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.022405    6506 scope.go:122] "RemoveContainer" containerID="94c0489fc4c6f194c5baf122536b4b4d41c49209483c84bdcb86b15aa4959887"
Nov 03 16:22:43.031945 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.031911    6506 scope.go:122] "RemoveContainer" containerID="94c0489fc4c6f194c5baf122536b4b4d41c49209483c84bdcb86b15aa4959887"
Nov 03 16:22:43.032187 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:43.032167    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c0489fc4c6f194c5baf122536b4b4d41c49209483c84bdcb86b15aa4959887\": container with ID starting with 94c0489fc4c6f194c5baf122536b4b4d41c49209483c84bdcb86b15aa4959887 not found: ID does not exist" containerID="94c0489fc4c6f194c5baf122536b4b4d41c49209483c84bdcb86b15aa4959887"
Nov 03 16:22:43.032233 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.032193    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c0489fc4c6f194c5baf122536b4b4d41c49209483c84bdcb86b15aa4959887"} err="failed to get container status \"94c0489fc4c6f194c5baf122536b4b4d41c49209483c84bdcb86b15aa4959887\": rpc error: code = NotFound desc = could not find container \"94c0489fc4c6f194c5baf122536b4b4d41c49209483c84bdcb86b15aa4959887\": container with ID starting with 94c0489fc4c6f194c5baf122536b4b4d41c49209483c84bdcb86b15aa4959887 not found: ID does not exist"
Nov 03 16:22:43.032233 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.032209    6506 scope.go:122] "RemoveContainer" containerID="6af2f3ffed58b75a271a821521541f45e11b196e361df4762fd1995408c88343"
Nov 03 16:22:43.040887 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.040872    6506 scope.go:122] "RemoveContainer" containerID="6af2f3ffed58b75a271a821521541f45e11b196e361df4762fd1995408c88343"
Nov 03 16:22:43.041165 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:43.041146    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af2f3ffed58b75a271a821521541f45e11b196e361df4762fd1995408c88343\": container with ID starting with 6af2f3ffed58b75a271a821521541f45e11b196e361df4762fd1995408c88343 not found: ID does not exist" containerID="6af2f3ffed58b75a271a821521541f45e11b196e361df4762fd1995408c88343"
Nov 03 16:22:43.041224 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.041172    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af2f3ffed58b75a271a821521541f45e11b196e361df4762fd1995408c88343"} err="failed to get container status \"6af2f3ffed58b75a271a821521541f45e11b196e361df4762fd1995408c88343\": rpc error: code = NotFound desc = could not find container \"6af2f3ffed58b75a271a821521541f45e11b196e361df4762fd1995408c88343\": container with ID starting with 6af2f3ffed58b75a271a821521541f45e11b196e361df4762fd1995408c88343 not found: ID does not exist"
Nov 03 16:22:43.055893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.055872    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"]
Nov 03 16:22:43.062091 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.062070    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-5mwlq"]
Nov 03 16:22:43.074318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.074292    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"]
Nov 03 16:22:43.079275 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.079256    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-2m894"]
Nov 03 16:22:43.094771 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.094749    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"]
Nov 03 16:22:43.099849 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.099827    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-wb4cc"]
Nov 03 16:22:43.587883 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.587850    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:43.760916 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:43.760888    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:44.101651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:44.101623    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27399426-17f7-4ea7-aba6-12338b82c6f3" path="/var/lib/kubelet/pods/27399426-17f7-4ea7-aba6-12338b82c6f3/volumes"
Nov 03 16:22:44.102174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:44.102149    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" path="/var/lib/kubelet/pods/b24bd837-1361-49af-bb52-dd51f01967a7/volumes"
Nov 03 16:22:44.102535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:44.102521    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" path="/var/lib/kubelet/pods/de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1/volumes"
Nov 03 16:22:44.428211 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:44.428135    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:44.691308 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:44.691235    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:46.980014 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:46.979934    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:47.194934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:47.194902    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:47.302760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:47.302731    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"]
Nov 03 16:22:47.302957 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:47.302937    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr" podUID="abd6d999-549b-4cbd-920a-39660ee9c30f" containerName="istio-proxy" containerID="cri-o://b7cc9d22c2b1c15109a5f732679e9ca916b81119e51ae158a8a7247f373c09a9" gracePeriod=30
Nov 03 16:22:47.331099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:47.331068    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"]
Nov 03 16:22:47.331286 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:47.331254    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt" podUID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerName="istio-proxy" containerID="cri-o://111d7eb51a6e97160d672581c5ec67686cf513d636ed2e4e9ad6c18aef11e9e1" gracePeriod=30
Nov 03 16:22:47.359069 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:47.358983    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"]
Nov 03 16:22:47.359511 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:47.359292    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs" podUID="063fc995-de52-43cf-a70f-0f2d163e015f" containerName="istio-proxy" containerID="cri-o://ccab854af11b6490a7106d30aba1e010efb884030e4e7add4c179586a2373f71" gracePeriod=30
Nov 03 16:22:47.401154 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:47.401088    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:48.299454 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:48.299418    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 03 16:22:48.299869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:48.299474    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr" podUID="abd6d999-549b-4cbd-920a-39660ee9c30f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:50.694931 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:50.694902    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:50.775355 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:50.775322    6506 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:50.775506 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:50.775377    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt" podUID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:50.867578 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:50.867544    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t" podUID="ccadaa1e-d59a-4231-ad4c-d8f46e7f4365" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:22:50.867749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:50.867545    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2" podUID="f5d53d56-ee8a-4465-8a5a-06e9064ab695" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:22:51.167410 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:51.167378    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:22:52.670839 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.670813    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:22:52.679719 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.679692    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-credential-socket\") pod \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") "
Nov 03 16:22:52.679865 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.679746    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-envoy\") pod \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") "
Nov 03 16:22:52.679865 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.679829    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-workload-socket\") pod \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") "
Nov 03 16:22:52.680002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.679908    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-workload-certs\") pod \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") "
Nov 03 16:22:52.680002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.679948    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-token\" (UniqueName: \"kubernetes.io/projected/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-token\") pod \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") "
Nov 03 16:22:52.680002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.679985    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-kube-api-access-px495\" (UniqueName: \"kubernetes.io/projected/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-kube-api-access-px495\") pod \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") "
Nov 03 16:22:52.680162 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.680015    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-credential-socket" pod "15f2a444-a2bf-46ed-9a2d-4a40dd09c537" (UID: "15f2a444-a2bf-46ed-9a2d-4a40dd09c537"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.680162 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.680034    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-data\") pod \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") "
Nov 03 16:22:52.680162 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.680068    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istiod-ca-cert\") pod \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") "
Nov 03 16:22:52.680162 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.680120    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-podinfo\") pod \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\" (UID: \"15f2a444-a2bf-46ed-9a2d-4a40dd09c537\") "
Nov 03 16:22:52.680372 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.680216    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-workload-certs" pod "15f2a444-a2bf-46ed-9a2d-4a40dd09c537" (UID: "15f2a444-a2bf-46ed-9a2d-4a40dd09c537"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.680454 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.680431    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-workload-socket" pod "15f2a444-a2bf-46ed-9a2d-4a40dd09c537" (UID: "15f2a444-a2bf-46ed-9a2d-4a40dd09c537"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.680512 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.680486    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.680512 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.680508    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.680825 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.680769    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-data" pod "15f2a444-a2bf-46ed-9a2d-4a40dd09c537" (UID: "15f2a444-a2bf-46ed-9a2d-4a40dd09c537"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.680988 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.680968    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istiod-ca-cert" pod "15f2a444-a2bf-46ed-9a2d-4a40dd09c537" (UID: "15f2a444-a2bf-46ed-9a2d-4a40dd09c537"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:52.683983 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.683962    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-envoy" pod "15f2a444-a2bf-46ed-9a2d-4a40dd09c537" (UID: "15f2a444-a2bf-46ed-9a2d-4a40dd09c537"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.684739 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.684715    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-podinfo" pod "15f2a444-a2bf-46ed-9a2d-4a40dd09c537" (UID: "15f2a444-a2bf-46ed-9a2d-4a40dd09c537"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:52.684739 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.684727    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-kube-api-access-px495" pod "15f2a444-a2bf-46ed-9a2d-4a40dd09c537" (UID: "15f2a444-a2bf-46ed-9a2d-4a40dd09c537"). InnerVolumeSpecName "kube-api-access-px495". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:52.684904 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.684752    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-token" pod "15f2a444-a2bf-46ed-9a2d-4a40dd09c537" (UID: "15f2a444-a2bf-46ed-9a2d-4a40dd09c537"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:52.726643 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.726295    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"
Nov 03 16:22:52.801152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.801107    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d53d56-ee8a-4465-8a5a-06e9064ab695-kube-api-access-tpfn9\" (UniqueName: \"kubernetes.io/projected/f5d53d56-ee8a-4465-8a5a-06e9064ab695-kube-api-access-tpfn9\") pod \"f5d53d56-ee8a-4465-8a5a-06e9064ab695\" (UID: \"f5d53d56-ee8a-4465-8a5a-06e9064ab695\") "
Nov 03 16:22:52.801339 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.801291    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d53d56-ee8a-4465-8a5a-06e9064ab695-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d53d56-ee8a-4465-8a5a-06e9064ab695-plugins-config-volume\") pod \"f5d53d56-ee8a-4465-8a5a-06e9064ab695\" (UID: \"f5d53d56-ee8a-4465-8a5a-06e9064ab695\") "
Nov 03 16:22:52.801618 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.801592    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.801700 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.801624    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px495\" (UniqueName: \"kubernetes.io/projected/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-kube-api-access-px495\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.801700 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.801640    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.801700 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.801654    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.801700 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.801668    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.801950 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.801700    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.801950 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.801713    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f2a444-a2bf-46ed-9a2d-4a40dd09c537-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.802124 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.802095    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d53d56-ee8a-4465-8a5a-06e9064ab695-plugins-config-volume" pod "f5d53d56-ee8a-4465-8a5a-06e9064ab695" (UID: "f5d53d56-ee8a-4465-8a5a-06e9064ab695"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:52.804515 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.804487    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d53d56-ee8a-4465-8a5a-06e9064ab695-kube-api-access-tpfn9" pod "f5d53d56-ee8a-4465-8a5a-06e9064ab695" (UID: "f5d53d56-ee8a-4465-8a5a-06e9064ab695"). InnerVolumeSpecName "kube-api-access-tpfn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:52.816646 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.816621    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"
Nov 03 16:22:52.819931 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.819912    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:22:52.828136 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.827638    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:22:52.902676 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.902604    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-workload-socket\") pod \"abd6d999-549b-4cbd-920a-39660ee9c30f\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") "
Nov 03 16:22:52.902676 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.902653    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/063fc995-de52-43cf-a70f-0f2d163e015f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063fc995-de52-43cf-a70f-0f2d163e015f-istiod-ca-cert\") pod \"063fc995-de52-43cf-a70f-0f2d163e015f\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") "
Nov 03 16:22:52.902676 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.902674    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063fc995-de52-43cf-a70f-0f2d163e015f-kube-api-access-zp7ps\" (UniqueName: \"kubernetes.io/projected/063fc995-de52-43cf-a70f-0f2d163e015f-kube-api-access-zp7ps\") pod \"063fc995-de52-43cf-a70f-0f2d163e015f\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") "
Nov 03 16:22:52.902997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.902692    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-workload-certs\") pod \"063fc995-de52-43cf-a70f-0f2d163e015f\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") "
Nov 03 16:22:52.902997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.902713    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-data\") pod \"abd6d999-549b-4cbd-920a-39660ee9c30f\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") "
Nov 03 16:22:52.902997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.902871    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063fc995-de52-43cf-a70f-0f2d163e015f-istio-token\" (UniqueName: \"kubernetes.io/projected/063fc995-de52-43cf-a70f-0f2d163e015f-istio-token\") pod \"063fc995-de52-43cf-a70f-0f2d163e015f\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") "
Nov 03 16:22:52.902997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.902915    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-workload-socket" pod "abd6d999-549b-4cbd-920a-39660ee9c30f" (UID: "abd6d999-549b-4cbd-920a-39660ee9c30f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.902997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.902951    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd6d999-549b-4cbd-920a-39660ee9c30f-kube-api-access-nc87k\" (UniqueName: \"kubernetes.io/projected/abd6d999-549b-4cbd-920a-39660ee9c30f-kube-api-access-nc87k\") pod \"abd6d999-549b-4cbd-920a-39660ee9c30f\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") "
Nov 03 16:22:52.902997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.902980    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-data" pod "abd6d999-549b-4cbd-920a-39660ee9c30f" (UID: "abd6d999-549b-4cbd-920a-39660ee9c30f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.903304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903007    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccadaa1e-d59a-4231-ad4c-d8f46e7f4365-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccadaa1e-d59a-4231-ad4c-d8f46e7f4365-plugins-config-volume\") pod \"ccadaa1e-d59a-4231-ad4c-d8f46e7f4365\" (UID: \"ccadaa1e-d59a-4231-ad4c-d8f46e7f4365\") "
Nov 03 16:22:52.903304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903044    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-envoy\") pod \"abd6d999-549b-4cbd-920a-39660ee9c30f\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") "
Nov 03 16:22:52.903304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903078    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-istio-envoy\") pod \"063fc995-de52-43cf-a70f-0f2d163e015f\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") "
Nov 03 16:22:52.903304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903103    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063fc995-de52-43cf-a70f-0f2d163e015f-istiod-ca-cert" pod "063fc995-de52-43cf-a70f-0f2d163e015f" (UID: "063fc995-de52-43cf-a70f-0f2d163e015f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:52.903304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903112    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-istio-data\") pod \"063fc995-de52-43cf-a70f-0f2d163e015f\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") "
Nov 03 16:22:52.903304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903182    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-workload-socket\") pod \"063fc995-de52-43cf-a70f-0f2d163e015f\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") "
Nov 03 16:22:52.903304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903300    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-istio-data" pod "063fc995-de52-43cf-a70f-0f2d163e015f" (UID: "063fc995-de52-43cf-a70f-0f2d163e015f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.903671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903320    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-workload-certs" pod "063fc995-de52-43cf-a70f-0f2d163e015f" (UID: "063fc995-de52-43cf-a70f-0f2d163e015f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.903671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903359    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-credential-socket\") pod \"063fc995-de52-43cf-a70f-0f2d163e015f\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") "
Nov 03 16:22:52.903671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903394    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-credential-socket\") pod \"abd6d999-549b-4cbd-920a-39660ee9c30f\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") "
Nov 03 16:22:52.903671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903420    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccadaa1e-d59a-4231-ad4c-d8f46e7f4365-plugins-config-volume" pod "ccadaa1e-d59a-4231-ad4c-d8f46e7f4365" (UID: "ccadaa1e-d59a-4231-ad4c-d8f46e7f4365"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:52.903671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903439    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-workload-certs\") pod \"abd6d999-549b-4cbd-920a-39660ee9c30f\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") "
Nov 03 16:22:52.903671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903498    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd6d999-549b-4cbd-920a-39660ee9c30f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd6d999-549b-4cbd-920a-39660ee9c30f-istiod-ca-cert\") pod \"abd6d999-549b-4cbd-920a-39660ee9c30f\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") "
Nov 03 16:22:52.903671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903531    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccadaa1e-d59a-4231-ad4c-d8f46e7f4365-kube-api-access-ns5kx\" (UniqueName: \"kubernetes.io/projected/ccadaa1e-d59a-4231-ad4c-d8f46e7f4365-kube-api-access-ns5kx\") pod \"ccadaa1e-d59a-4231-ad4c-d8f46e7f4365\" (UID: \"ccadaa1e-d59a-4231-ad4c-d8f46e7f4365\") "
Nov 03 16:22:52.903671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903592    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-token\" (UniqueName: \"kubernetes.io/projected/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-token\") pod \"abd6d999-549b-4cbd-920a-39660ee9c30f\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") "
Nov 03 16:22:52.903671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903599    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-workload-socket" pod "063fc995-de52-43cf-a70f-0f2d163e015f" (UID: "063fc995-de52-43cf-a70f-0f2d163e015f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.903671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903633    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/063fc995-de52-43cf-a70f-0f2d163e015f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063fc995-de52-43cf-a70f-0f2d163e015f-istio-podinfo\") pod \"063fc995-de52-43cf-a70f-0f2d163e015f\" (UID: \"063fc995-de52-43cf-a70f-0f2d163e015f\") "
Nov 03 16:22:52.903671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903670    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-podinfo\") pod \"abd6d999-549b-4cbd-920a-39660ee9c30f\" (UID: \"abd6d999-549b-4cbd-920a-39660ee9c30f\") "
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903782    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-workload-certs" pod "abd6d999-549b-4cbd-920a-39660ee9c30f" (UID: "abd6d999-549b-4cbd-920a-39660ee9c30f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903907    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-credential-socket" pod "063fc995-de52-43cf-a70f-0f2d163e015f" (UID: "063fc995-de52-43cf-a70f-0f2d163e015f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.903992    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-credential-socket" pod "abd6d999-549b-4cbd-920a-39660ee9c30f" (UID: "abd6d999-549b-4cbd-920a-39660ee9c30f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.904127    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpfn9\" (UniqueName: \"kubernetes.io/projected/f5d53d56-ee8a-4465-8a5a-06e9064ab695-kube-api-access-tpfn9\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.904150    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.904167    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063fc995-de52-43cf-a70f-0f2d163e015f-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.904181    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.904196    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.904210    6506 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccadaa1e-d59a-4231-ad4c-d8f46e7f4365-plugins-config-volume\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.904214    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd6d999-549b-4cbd-920a-39660ee9c30f-istiod-ca-cert" pod "abd6d999-549b-4cbd-920a-39660ee9c30f" (UID: "abd6d999-549b-4cbd-920a-39660ee9c30f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.904224    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.904256    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.904274    6506 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d53d56-ee8a-4465-8a5a-06e9064ab695-plugins-config-volume\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.904290    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.904304    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.904322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.904318    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.906319 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.905897    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063fc995-de52-43cf-a70f-0f2d163e015f-istio-token" pod "063fc995-de52-43cf-a70f-0f2d163e015f" (UID: "063fc995-de52-43cf-a70f-0f2d163e015f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:52.906319 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.906258    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-envoy" pod "abd6d999-549b-4cbd-920a-39660ee9c30f" (UID: "abd6d999-549b-4cbd-920a-39660ee9c30f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.906755 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.906721    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd6d999-549b-4cbd-920a-39660ee9c30f-kube-api-access-nc87k" pod "abd6d999-549b-4cbd-920a-39660ee9c30f" (UID: "abd6d999-549b-4cbd-920a-39660ee9c30f"). InnerVolumeSpecName "kube-api-access-nc87k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:52.907043 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.907019    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063fc995-de52-43cf-a70f-0f2d163e015f-kube-api-access-zp7ps" pod "063fc995-de52-43cf-a70f-0f2d163e015f" (UID: "063fc995-de52-43cf-a70f-0f2d163e015f"). InnerVolumeSpecName "kube-api-access-zp7ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:52.907304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.907283    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccadaa1e-d59a-4231-ad4c-d8f46e7f4365-kube-api-access-ns5kx" pod "ccadaa1e-d59a-4231-ad4c-d8f46e7f4365" (UID: "ccadaa1e-d59a-4231-ad4c-d8f46e7f4365"). InnerVolumeSpecName "kube-api-access-ns5kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:52.907778 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.907745    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-token" pod "abd6d999-549b-4cbd-920a-39660ee9c30f" (UID: "abd6d999-549b-4cbd-920a-39660ee9c30f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:52.907906 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.907778    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-istio-envoy" pod "063fc995-de52-43cf-a70f-0f2d163e015f" (UID: "063fc995-de52-43cf-a70f-0f2d163e015f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.908110 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.908082    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-podinfo" pod "abd6d999-549b-4cbd-920a-39660ee9c30f" (UID: "abd6d999-549b-4cbd-920a-39660ee9c30f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:52.917963 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:52.917934    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/063fc995-de52-43cf-a70f-0f2d163e015f-istio-podinfo" pod "063fc995-de52-43cf-a70f-0f2d163e015f" (UID: "063fc995-de52-43cf-a70f-0f2d163e015f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:53.005035 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.004997    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.005035 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.005034    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063fc995-de52-43cf-a70f-0f2d163e015f-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.005249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.005048    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.005249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.005061    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp7ps\" (UniqueName: \"kubernetes.io/projected/063fc995-de52-43cf-a70f-0f2d163e015f-kube-api-access-zp7ps\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.005249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.005073    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063fc995-de52-43cf-a70f-0f2d163e015f-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.005249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.005086    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc87k\" (UniqueName: \"kubernetes.io/projected/abd6d999-549b-4cbd-920a-39660ee9c30f-kube-api-access-nc87k\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.005249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.005098    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd6d999-549b-4cbd-920a-39660ee9c30f-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.005249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.005109    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063fc995-de52-43cf-a70f-0f2d163e015f-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.005249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.005129    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd6d999-549b-4cbd-920a-39660ee9c30f-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.005249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.005143    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns5kx\" (UniqueName: \"kubernetes.io/projected/ccadaa1e-d59a-4231-ad4c-d8f46e7f4365-kube-api-access-ns5kx\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.084369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.084335    6506 generic.go:356] "Generic (PLEG): container finished" podID="063fc995-de52-43cf-a70f-0f2d163e015f" containerID="ccab854af11b6490a7106d30aba1e010efb884030e4e7add4c179586a2373f71" exitCode=0
Nov 03 16:22:53.084552 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.084386    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"
Nov 03 16:22:53.084552 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.084425    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs" event={"ID":"063fc995-de52-43cf-a70f-0f2d163e015f","Type":"ContainerDied","Data":"ccab854af11b6490a7106d30aba1e010efb884030e4e7add4c179586a2373f71"}
Nov 03 16:22:53.084552 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.084512    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs" event={"ID":"063fc995-de52-43cf-a70f-0f2d163e015f","Type":"ContainerDied","Data":"d891ed9597a3948b6972e8ba371d33ced9bc9776fa9c67616bc6d6d68037b413"}
Nov 03 16:22:53.084552 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.084545    6506 scope.go:122] "RemoveContainer" containerID="ccab854af11b6490a7106d30aba1e010efb884030e4e7add4c179586a2373f71"
Nov 03 16:22:53.085851 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.085829    6506 generic.go:356] "Generic (PLEG): container finished" podID="f5d53d56-ee8a-4465-8a5a-06e9064ab695" containerID="c294d8fa8288047d98fbfa384ed565250c49758c4b9a7873ea128d30aeaa021b" exitCode=0
Nov 03 16:22:53.086032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.085932    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"
Nov 03 16:22:53.086032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.085936    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2" event={"ID":"f5d53d56-ee8a-4465-8a5a-06e9064ab695","Type":"ContainerDied","Data":"c294d8fa8288047d98fbfa384ed565250c49758c4b9a7873ea128d30aeaa021b"}
Nov 03 16:22:53.086136 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.086068    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2" event={"ID":"f5d53d56-ee8a-4465-8a5a-06e9064ab695","Type":"ContainerDied","Data":"6d37e37ef0946caf88a753638c64f568eec54d050f1126fe06b533b602c78060"}
Nov 03 16:22:53.087481 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.087459    6506 generic.go:356] "Generic (PLEG): container finished" podID="abd6d999-549b-4cbd-920a-39660ee9c30f" containerID="b7cc9d22c2b1c15109a5f732679e9ca916b81119e51ae158a8a7247f373c09a9" exitCode=0
Nov 03 16:22:53.087610 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.087486    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr" event={"ID":"abd6d999-549b-4cbd-920a-39660ee9c30f","Type":"ContainerDied","Data":"b7cc9d22c2b1c15109a5f732679e9ca916b81119e51ae158a8a7247f373c09a9"}
Nov 03 16:22:53.087610 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.087514    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"
Nov 03 16:22:53.087610 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.087524    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr" event={"ID":"abd6d999-549b-4cbd-920a-39660ee9c30f","Type":"ContainerDied","Data":"c82ca920fbae28952fdf4da0a826bd0f8e6a98eda5412056e62dbc6b0783a244"}
Nov 03 16:22:53.090099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.089633    6506 generic.go:356] "Generic (PLEG): container finished" podID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerID="111d7eb51a6e97160d672581c5ec67686cf513d636ed2e4e9ad6c18aef11e9e1" exitCode=0
Nov 03 16:22:53.090099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.089681    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"
Nov 03 16:22:53.090099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.089714    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt" event={"ID":"15f2a444-a2bf-46ed-9a2d-4a40dd09c537","Type":"ContainerDied","Data":"111d7eb51a6e97160d672581c5ec67686cf513d636ed2e4e9ad6c18aef11e9e1"}
Nov 03 16:22:53.090099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.089756    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt" event={"ID":"15f2a444-a2bf-46ed-9a2d-4a40dd09c537","Type":"ContainerDied","Data":"94a8eee9dcb8a79512a24572a093bcfe0adff24b4b5a257620e7daa146d770bb"}
Nov 03 16:22:53.091590 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.091570    6506 generic.go:356] "Generic (PLEG): container finished" podID="ccadaa1e-d59a-4231-ad4c-d8f46e7f4365" containerID="e1f0b88be24b3429927184dd1f7e14df0d93aa3ba1a9ee036d331725d9fa1cad" exitCode=0
Nov 03 16:22:53.091703 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.091602    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t" event={"ID":"ccadaa1e-d59a-4231-ad4c-d8f46e7f4365","Type":"ContainerDied","Data":"e1f0b88be24b3429927184dd1f7e14df0d93aa3ba1a9ee036d331725d9fa1cad"}
Nov 03 16:22:53.091703 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.091627    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t" event={"ID":"ccadaa1e-d59a-4231-ad4c-d8f46e7f4365","Type":"ContainerDied","Data":"3a2fd32ddb3707137d90fa650da157c72c4deb162c08bb9117018120c933a177"}
Nov 03 16:22:53.091703 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.091671    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"
Nov 03 16:22:53.097649 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.097545    6506 scope.go:122] "RemoveContainer" containerID="ccab854af11b6490a7106d30aba1e010efb884030e4e7add4c179586a2373f71"
Nov 03 16:22:53.097915 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:53.097886    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccab854af11b6490a7106d30aba1e010efb884030e4e7add4c179586a2373f71\": container with ID starting with ccab854af11b6490a7106d30aba1e010efb884030e4e7add4c179586a2373f71 not found: ID does not exist" containerID="ccab854af11b6490a7106d30aba1e010efb884030e4e7add4c179586a2373f71"
Nov 03 16:22:53.098007 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.097927    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccab854af11b6490a7106d30aba1e010efb884030e4e7add4c179586a2373f71"} err="failed to get container status \"ccab854af11b6490a7106d30aba1e010efb884030e4e7add4c179586a2373f71\": rpc error: code = NotFound desc = could not find container \"ccab854af11b6490a7106d30aba1e010efb884030e4e7add4c179586a2373f71\": container with ID starting with ccab854af11b6490a7106d30aba1e010efb884030e4e7add4c179586a2373f71 not found: ID does not exist"
Nov 03 16:22:53.098007 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.097952    6506 scope.go:122] "RemoveContainer" containerID="c294d8fa8288047d98fbfa384ed565250c49758c4b9a7873ea128d30aeaa021b"
Nov 03 16:22:53.110666 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.110551    6506 scope.go:122] "RemoveContainer" containerID="c294d8fa8288047d98fbfa384ed565250c49758c4b9a7873ea128d30aeaa021b"
Nov 03 16:22:53.110955 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:53.110930    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c294d8fa8288047d98fbfa384ed565250c49758c4b9a7873ea128d30aeaa021b\": container with ID starting with c294d8fa8288047d98fbfa384ed565250c49758c4b9a7873ea128d30aeaa021b not found: ID does not exist" containerID="c294d8fa8288047d98fbfa384ed565250c49758c4b9a7873ea128d30aeaa021b"
Nov 03 16:22:53.111043 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.110965    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c294d8fa8288047d98fbfa384ed565250c49758c4b9a7873ea128d30aeaa021b"} err="failed to get container status \"c294d8fa8288047d98fbfa384ed565250c49758c4b9a7873ea128d30aeaa021b\": rpc error: code = NotFound desc = could not find container \"c294d8fa8288047d98fbfa384ed565250c49758c4b9a7873ea128d30aeaa021b\": container with ID starting with c294d8fa8288047d98fbfa384ed565250c49758c4b9a7873ea128d30aeaa021b not found: ID does not exist"
Nov 03 16:22:53.111043 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.110987    6506 scope.go:122] "RemoveContainer" containerID="b7cc9d22c2b1c15109a5f732679e9ca916b81119e51ae158a8a7247f373c09a9"
Nov 03 16:22:53.122242 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.122211    6506 scope.go:122] "RemoveContainer" containerID="b7cc9d22c2b1c15109a5f732679e9ca916b81119e51ae158a8a7247f373c09a9"
Nov 03 16:22:53.122565 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:53.122538    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7cc9d22c2b1c15109a5f732679e9ca916b81119e51ae158a8a7247f373c09a9\": container with ID starting with b7cc9d22c2b1c15109a5f732679e9ca916b81119e51ae158a8a7247f373c09a9 not found: ID does not exist" containerID="b7cc9d22c2b1c15109a5f732679e9ca916b81119e51ae158a8a7247f373c09a9"
Nov 03 16:22:53.122662 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.122568    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7cc9d22c2b1c15109a5f732679e9ca916b81119e51ae158a8a7247f373c09a9"} err="failed to get container status \"b7cc9d22c2b1c15109a5f732679e9ca916b81119e51ae158a8a7247f373c09a9\": rpc error: code = NotFound desc = could not find container \"b7cc9d22c2b1c15109a5f732679e9ca916b81119e51ae158a8a7247f373c09a9\": container with ID starting with b7cc9d22c2b1c15109a5f732679e9ca916b81119e51ae158a8a7247f373c09a9 not found: ID does not exist"
Nov 03 16:22:53.122662 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.122595    6506 scope.go:122] "RemoveContainer" containerID="111d7eb51a6e97160d672581c5ec67686cf513d636ed2e4e9ad6c18aef11e9e1"
Nov 03 16:22:53.126527 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.126502    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"]
Nov 03 16:22:53.134680 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.134486    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qx4xr"]
Nov 03 16:22:53.139103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.138686    6506 scope.go:122] "RemoveContainer" containerID="cd4df6ddd25a97f54b079e7943d6c6c6791ad373e3e2727e1f3550a0f0c01aab"
Nov 03 16:22:53.149863 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.149749    6506 scope.go:122] "RemoveContainer" containerID="111d7eb51a6e97160d672581c5ec67686cf513d636ed2e4e9ad6c18aef11e9e1"
Nov 03 16:22:53.151239 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:53.151097    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111d7eb51a6e97160d672581c5ec67686cf513d636ed2e4e9ad6c18aef11e9e1\": container with ID starting with 111d7eb51a6e97160d672581c5ec67686cf513d636ed2e4e9ad6c18aef11e9e1 not found: ID does not exist" containerID="111d7eb51a6e97160d672581c5ec67686cf513d636ed2e4e9ad6c18aef11e9e1"
Nov 03 16:22:53.151239 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.151132    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111d7eb51a6e97160d672581c5ec67686cf513d636ed2e4e9ad6c18aef11e9e1"} err="failed to get container status \"111d7eb51a6e97160d672581c5ec67686cf513d636ed2e4e9ad6c18aef11e9e1\": rpc error: code = NotFound desc = could not find container \"111d7eb51a6e97160d672581c5ec67686cf513d636ed2e4e9ad6c18aef11e9e1\": container with ID starting with 111d7eb51a6e97160d672581c5ec67686cf513d636ed2e4e9ad6c18aef11e9e1 not found: ID does not exist"
Nov 03 16:22:53.151239 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.151160    6506 scope.go:122] "RemoveContainer" containerID="cd4df6ddd25a97f54b079e7943d6c6c6791ad373e3e2727e1f3550a0f0c01aab"
Nov 03 16:22:53.151470 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:53.151447    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4df6ddd25a97f54b079e7943d6c6c6791ad373e3e2727e1f3550a0f0c01aab\": container with ID starting with cd4df6ddd25a97f54b079e7943d6c6c6791ad373e3e2727e1f3550a0f0c01aab not found: ID does not exist" containerID="cd4df6ddd25a97f54b079e7943d6c6c6791ad373e3e2727e1f3550a0f0c01aab"
Nov 03 16:22:53.151516 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.151475    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4df6ddd25a97f54b079e7943d6c6c6791ad373e3e2727e1f3550a0f0c01aab"} err="failed to get container status \"cd4df6ddd25a97f54b079e7943d6c6c6791ad373e3e2727e1f3550a0f0c01aab\": rpc error: code = NotFound desc = could not find container \"cd4df6ddd25a97f54b079e7943d6c6c6791ad373e3e2727e1f3550a0f0c01aab\": container with ID starting with cd4df6ddd25a97f54b079e7943d6c6c6791ad373e3e2727e1f3550a0f0c01aab not found: ID does not exist"
Nov 03 16:22:53.151516 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.151501    6506 scope.go:122] "RemoveContainer" containerID="e1f0b88be24b3429927184dd1f7e14df0d93aa3ba1a9ee036d331725d9fa1cad"
Nov 03 16:22:53.155235 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.155182    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"]
Nov 03 16:22:53.162060 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.162041    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-d92dt"]
Nov 03 16:22:53.167286 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.167259    6506 scope.go:122] "RemoveContainer" containerID="e1f0b88be24b3429927184dd1f7e14df0d93aa3ba1a9ee036d331725d9fa1cad"
Nov 03 16:22:53.167624 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:53.167588    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f0b88be24b3429927184dd1f7e14df0d93aa3ba1a9ee036d331725d9fa1cad\": container with ID starting with e1f0b88be24b3429927184dd1f7e14df0d93aa3ba1a9ee036d331725d9fa1cad not found: ID does not exist" containerID="e1f0b88be24b3429927184dd1f7e14df0d93aa3ba1a9ee036d331725d9fa1cad"
Nov 03 16:22:53.167707 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.167635    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f0b88be24b3429927184dd1f7e14df0d93aa3ba1a9ee036d331725d9fa1cad"} err="failed to get container status \"e1f0b88be24b3429927184dd1f7e14df0d93aa3ba1a9ee036d331725d9fa1cad\": rpc error: code = NotFound desc = could not find container \"e1f0b88be24b3429927184dd1f7e14df0d93aa3ba1a9ee036d331725d9fa1cad\": container with ID starting with e1f0b88be24b3429927184dd1f7e14df0d93aa3ba1a9ee036d331725d9fa1cad not found: ID does not exist"
Nov 03 16:22:53.178305 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.178282    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"]
Nov 03 16:22:53.182218 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.182198    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b132e3-68c6b955f9-8cp7t"]
Nov 03 16:22:53.198832 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.198809    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"]
Nov 03 16:22:53.204406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.204385    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-twvvs"]
Nov 03 16:22:53.216769 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.216724    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"]
Nov 03 16:22:53.220080 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:53.220060    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b132e3-7cf65db647-q2sp2"]
Nov 03 16:22:54.102203 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:54.102166    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="063fc995-de52-43cf-a70f-0f2d163e015f" path="/var/lib/kubelet/pods/063fc995-de52-43cf-a70f-0f2d163e015f/volumes"
Nov 03 16:22:54.102718 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:54.102699    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" path="/var/lib/kubelet/pods/15f2a444-a2bf-46ed-9a2d-4a40dd09c537/volumes"
Nov 03 16:22:54.103194 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:54.103179    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd6d999-549b-4cbd-920a-39660ee9c30f" path="/var/lib/kubelet/pods/abd6d999-549b-4cbd-920a-39660ee9c30f/volumes"
Nov 03 16:22:54.103783 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:54.103769    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccadaa1e-d59a-4231-ad4c-d8f46e7f4365" path="/var/lib/kubelet/pods/ccadaa1e-d59a-4231-ad4c-d8f46e7f4365/volumes"
Nov 03 16:22:54.104128 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:22:54.104115    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d53d56-ee8a-4465-8a5a-06e9064ab695" path="/var/lib/kubelet/pods/f5d53d56-ee8a-4465-8a5a-06e9064ab695/volumes"
Nov 03 16:22:58.795988 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:22:58.795951    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:23:07.004628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.004595    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"]
Nov 03 16:23:07.005146 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005065    6506 cpu_manager.go:414] "containerMap: removing container" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy"
Nov 03 16:23:07.005146 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005083    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy"
Nov 03 16:23:07.005146 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005096    6506 cpu_manager.go:414] "containerMap: removing container" podUID="27399426-17f7-4ea7-aba6-12338b82c6f3" containerName="istio-proxy"
Nov 03 16:23:07.005146 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005103    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27399426-17f7-4ea7-aba6-12338b82c6f3" containerName="istio-proxy"
Nov 03 16:23:07.005146 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005142    6506 cpu_manager.go:414] "containerMap: removing container" podUID="af0fa8f4-44df-45e7-8028-7d329877afaa" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005149    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0fa8f4-44df-45e7-8028-7d329877afaa" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005162    6506 cpu_manager.go:414] "containerMap: removing container" podUID="119e87d8-1b7b-46db-9ac8-289804f75184" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005169    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119e87d8-1b7b-46db-9ac8-289804f75184" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005180    6506 cpu_manager.go:414] "containerMap: removing container" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005186    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005199    6506 cpu_manager.go:414] "containerMap: removing container" podUID="986e48d0-8cf9-45f3-86c7-06511006a2e1" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005206    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986e48d0-8cf9-45f3-86c7-06511006a2e1" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005218    6506 cpu_manager.go:414] "containerMap: removing container" podUID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005223    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005230    6506 cpu_manager.go:414] "containerMap: removing container" podUID="063fc995-de52-43cf-a70f-0f2d163e015f" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005234    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="063fc995-de52-43cf-a70f-0f2d163e015f" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005241    6506 cpu_manager.go:414] "containerMap: removing container" podUID="e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005244    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005251    6506 cpu_manager.go:414] "containerMap: removing container" podUID="39813549-9eb7-4cb7-9f08-4bf983b4cebc" containerName="payload-processing"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005255    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39813549-9eb7-4cb7-9f08-4bf983b4cebc" containerName="payload-processing"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005262    6506 cpu_manager.go:414] "containerMap: removing container" podUID="abd6d999-549b-4cbd-920a-39660ee9c30f" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005267    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd6d999-549b-4cbd-920a-39660ee9c30f" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005281    6506 cpu_manager.go:414] "containerMap: removing container" podUID="20b9b08d-2d97-4604-890c-10f6305c0108" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005286    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b9b08d-2d97-4604-890c-10f6305c0108" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005292    6506 cpu_manager.go:414] "containerMap: removing container" podUID="17b47389-9b79-402f-bc9b-d404e0b1e75c" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005296    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b47389-9b79-402f-bc9b-d404e0b1e75c" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005303    6506 cpu_manager.go:414] "containerMap: removing container" podUID="ccadaa1e-d59a-4231-ad4c-d8f46e7f4365" containerName="payload-pre-processing"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005307    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccadaa1e-d59a-4231-ad4c-d8f46e7f4365" containerName="payload-pre-processing"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005313    6506 cpu_manager.go:414] "containerMap: removing container" podUID="20b9b08d-2d97-4604-890c-10f6305c0108" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005318    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b9b08d-2d97-4604-890c-10f6305c0108" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005336    6506 cpu_manager.go:414] "containerMap: removing container" podUID="5a76c7da-5ade-41c8-a075-430ccf3e7215" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005340    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a76c7da-5ade-41c8-a075-430ccf3e7215" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005347    6506 cpu_manager.go:414] "containerMap: removing container" podUID="de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005351    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005362    6506 cpu_manager.go:414] "containerMap: removing container" podUID="f5d53d56-ee8a-4465-8a5a-06e9064ab695" containerName="payload-processing"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005366    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d53d56-ee8a-4465-8a5a-06e9064ab695" containerName="payload-processing"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005372    6506 cpu_manager.go:414] "containerMap: removing container" podUID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerName="istio-proxy"
Nov 03 16:23:07.005361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005376    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerName="istio-proxy"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005384    6506 cpu_manager.go:414] "containerMap: removing container" podUID="f093183a-3336-4dce-8945-f907211722f4" containerName="revoke-keys"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005388    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f093183a-3336-4dce-8945-f907211722f4" containerName="revoke-keys"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005470    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d53d56-ee8a-4465-8a5a-06e9064ab695" containerName="payload-processing"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005477    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerName="istio-proxy"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005484    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="f093183a-3336-4dce-8945-f907211722f4" containerName="revoke-keys"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005490    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24bd837-1361-49af-bb52-dd51f01967a7" containerName="istio-proxy"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005495    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="27399426-17f7-4ea7-aba6-12338b82c6f3" containerName="istio-proxy"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005501    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0fa8f4-44df-45e7-8028-7d329877afaa" containerName="istio-proxy"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005506    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="119e87d8-1b7b-46db-9ac8-289804f75184" containerName="istio-proxy"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005510    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7fb2203-adc1-4dcb-b092-42f6d5296165" containerName="main"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005516    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="986e48d0-8cf9-45f3-86c7-06511006a2e1" containerName="istio-proxy"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005521    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="063fc995-de52-43cf-a70f-0f2d163e015f" containerName="istio-proxy"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005526    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42754b4-ae9a-41b2-b4e4-0dad7d75a2f3" containerName="istio-proxy"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005531    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="39813549-9eb7-4cb7-9f08-4bf983b4cebc" containerName="payload-processing"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005536    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd6d999-549b-4cbd-920a-39660ee9c30f" containerName="istio-proxy"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005543    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b9b08d-2d97-4604-890c-10f6305c0108" containerName="istio-proxy"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005547    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="17b47389-9b79-402f-bc9b-d404e0b1e75c" containerName="istio-proxy"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005552    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccadaa1e-d59a-4231-ad4c-d8f46e7f4365" containerName="payload-pre-processing"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005556    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b9b08d-2d97-4604-890c-10f6305c0108" containerName="istio-proxy"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005560    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a76c7da-5ade-41c8-a075-430ccf3e7215" containerName="istio-proxy"
Nov 03 16:23:07.006340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.005565    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="de9d0efd-35df-4e0e-a9fb-dff1b6ca36c1" containerName="istio-proxy"
Nov 03 16:23:07.012182 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.012158    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.038889 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.038860    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"]
Nov 03 16:23:07.041227 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.041205    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"]
Nov 03 16:23:07.041631 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.041615    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f2a444-a2bf-46ed-9a2d-4a40dd09c537" containerName="istio-proxy"
Nov 03 16:23:07.047012 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.046988    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"]
Nov 03 16:23:07.047138 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.047075    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.144933 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.144897    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.145103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.144939    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.145103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.144964    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.145103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.145038    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.145103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.145083    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.145249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.145169    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.145249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.145200    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df80d95-49e6-476a-a626-0dd06e6afb66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.145249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.145219    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.145339 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.145255    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.145339 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.145289    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.145339 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.145310    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.145433 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.145373    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.145433 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.145390    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.145433 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.145416    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65nj4\" (UniqueName: \"kubernetes.io/projected/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-kube-api-access-65nj4\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.145588 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.145455    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8pn9\" (UniqueName: \"kubernetes.io/projected/9df80d95-49e6-476a-a626-0dd06e6afb66-kube-api-access-p8pn9\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.145588 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.145548    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.145588 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.145581    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.145703 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.145607    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.246783 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.246752    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.246957 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.246827    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65nj4\" (UniqueName: \"kubernetes.io/projected/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-kube-api-access-65nj4\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.246957 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.246878    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8pn9\" (UniqueName: \"kubernetes.io/projected/9df80d95-49e6-476a-a626-0dd06e6afb66-kube-api-access-p8pn9\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.246957 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.246927    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.246957 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.246951    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.247179 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.246975    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.247179 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247103    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.247179 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247155    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.247380 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247182    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.247380 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247209    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.247380 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247235    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.247380 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247281    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.247380 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247311    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df80d95-49e6-476a-a626-0dd06e6afb66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.247380 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247342    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.247380 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247338    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.247781 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247479    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.247781 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247573    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.247781 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247619    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.247970 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247780    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.247970 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247624    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.247970 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247875    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.247970 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247917    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.248159 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.247986    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df80d95-49e6-476a-a626-0dd06e6afb66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.248275 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.248249    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.248593 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.248309    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.248593 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.248384    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.248593 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.248511    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.248836 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.248643    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.249950 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.249926    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.250174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.250154    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.250315 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.250293    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.250359 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.250349    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.260101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.260018    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.260383 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.260359    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65nj4\" (UniqueName: \"kubernetes.io/projected/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-kube-api-access-65nj4\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-2prf7\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.260547 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.260451    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.260680 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.260658    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8pn9\" (UniqueName: \"kubernetes.io/projected/9df80d95-49e6-476a-a626-0dd06e6afb66-kube-api-access-p8pn9\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-n9mvg\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.329170 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.329147    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:07.362183 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.362149    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:07.616166 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.616136    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"]
Nov 03 16:23:07.617328 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:23:07.617303    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01649a37_cda3_4c59_8bd5_a3a1771d6fcc.slice/crio-dac9caecf09e81367d723f041e88c562756a4c32e94bd9c292d0016edaaaa280 WatchSource:0}: Error finding container dac9caecf09e81367d723f041e88c562756a4c32e94bd9c292d0016edaaaa280: Status 404 returned error can't find the container with id dac9caecf09e81367d723f041e88c562756a4c32e94bd9c292d0016edaaaa280
Nov 03 16:23:07.619409 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.619377    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:23:07.619542 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.619437    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:23:07.619542 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.619469    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:23:07.652576 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.652553    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"]
Nov 03 16:23:07.654952 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:23:07.654920    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df80d95_49e6_476a_a626_0dd06e6afb66.slice/crio-d7f875355354af0bbd923b6959f1467b29da5b66ff910f767243ab497d0edfa5 WatchSource:0}: Error finding container d7f875355354af0bbd923b6959f1467b29da5b66ff910f767243ab497d0edfa5: Status 404 returned error can't find the container with id d7f875355354af0bbd923b6959f1467b29da5b66ff910f767243ab497d0edfa5
Nov 03 16:23:07.657172 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.657141    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:23:07.657248 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.657202    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:23:07.657248 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:07.657230    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:23:08.194924 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:08.194883    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" event={"ID":"9df80d95-49e6-476a-a626-0dd06e6afb66","Type":"ContainerStarted","Data":"bc4429adad9428ed1eb2bda8566b67a254ccbd7aca5a1350e1ba74c096679a3c"}
Nov 03 16:23:08.195317 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:08.194960    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" event={"ID":"9df80d95-49e6-476a-a626-0dd06e6afb66","Type":"ContainerStarted","Data":"d7f875355354af0bbd923b6959f1467b29da5b66ff910f767243ab497d0edfa5"}
Nov 03 16:23:08.196276 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:08.196255    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" event={"ID":"01649a37-cda3-4c59-8bd5-a3a1771d6fcc","Type":"ContainerStarted","Data":"197f23e8960b82eae5097ce89bc5f2503f05db4727e6484631cdc6cdd358ad08"}
Nov 03 16:23:08.196323 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:08.196292    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" event={"ID":"01649a37-cda3-4c59-8bd5-a3a1771d6fcc","Type":"ContainerStarted","Data":"dac9caecf09e81367d723f041e88c562756a4c32e94bd9c292d0016edaaaa280"}
Nov 03 16:23:08.250819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:08.246628    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" podStartSLOduration=2.24660619 podStartE2EDuration="2.24660619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:23:08.226279887 +0000 UTC m=+1202.239678033" watchObservedRunningTime="2026-11-03 16:23:08.24660619 +0000 UTC m=+1202.260004337"
Nov 03 16:23:08.264445 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:08.264297    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" podStartSLOduration=2.2642774279999998 podStartE2EDuration="2.264277428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:23:08.262476474 +0000 UTC m=+1202.275874620" watchObservedRunningTime="2026-11-03 16:23:08.264277428 +0000 UTC m=+1202.277675576"
Nov 03 16:23:08.333269 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:08.333184    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:08.335701 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:08.335346    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-2prf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:23:08.335701 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:08.335403    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" podUID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 16:23:08.362388 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:08.362351    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:08.363841 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:08.363784    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 16:23:08.363956 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:08.363855    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 16:23:09.330821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:09.330167    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-2prf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:23:09.330821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:09.330230    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" podUID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 16:23:09.363506 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:09.363470    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 16:23:09.363682 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:09.363521    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 16:23:09.792512 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:09.792451    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:23:10.333819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:10.333209    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-2prf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:23:10.333819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:10.333269    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" podUID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 16:23:10.363587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:10.363552    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 16:23:10.363761 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:10.363605    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 16:23:11.336822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:11.333445    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-2prf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:23:11.336822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:11.333558    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" podUID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 16:23:11.363177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:11.363141    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 16:23:11.363463 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:11.363431    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 16:23:12.332181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:12.332143    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-2prf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:23:12.332474 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:12.332447    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" podUID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 16:23:12.363572 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:12.363529    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 16:23:12.363996 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:12.363600    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 16:23:13.287625 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.287593    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"]
Nov 03 16:23:13.292111 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.292090    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"
Nov 03 16:23:13.296307 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.296270    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:23:13.296447 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.296426    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:23:13.297660 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.297647    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:23:13.297805 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.297774    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:23:13.320302 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.320221    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"]
Nov 03 16:23:13.362128 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.362093    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-2prf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:23:13.362976 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.362363    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" podUID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 16:23:13.363116 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.363028    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 16:23:13.363116 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.363075    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 16:23:13.439581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.439541    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2727v\" (UID: \"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"
Nov 03 16:23:13.441048 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.441022    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjv5k\" (UniqueName: \"kubernetes.io/projected/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-kube-api-access-gjv5k\") pod \"maas-api-revoke-sub-85237e45-2727v\" (UID: \"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"
Nov 03 16:23:13.441247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.441229    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2727v\" (UID: \"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"
Nov 03 16:23:13.546516 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.546413    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjv5k\" (UniqueName: \"kubernetes.io/projected/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-kube-api-access-gjv5k\") pod \"maas-api-revoke-sub-85237e45-2727v\" (UID: \"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"
Nov 03 16:23:13.547234 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.547208    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2727v\" (UID: \"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"
Nov 03 16:23:13.548243 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.548224    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2727v\" (UID: \"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"
Nov 03 16:23:13.548974 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.548146    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2727v\" (UID: \"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"
Nov 03 16:23:13.553110 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.553087    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2727v\" (UID: \"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"
Nov 03 16:23:13.559904 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.559883    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjv5k\" (UniqueName: \"kubernetes.io/projected/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-kube-api-access-gjv5k\") pod \"maas-api-revoke-sub-85237e45-2727v\" (UID: \"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"
Nov 03 16:23:13.661589 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.661546    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"
Nov 03 16:23:13.953337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:13.953248    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"]
Nov 03 16:23:13.957007 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:23:13.956969    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa8e4f0_6e43_4cd9_9cb2_bfbd55627b17.slice/crio-1c51e13f97f97b4bea36b3a2eca6b1e7c576edd1c7c38e79bd9d813c521e9325 WatchSource:0}: Error finding container 1c51e13f97f97b4bea36b3a2eca6b1e7c576edd1c7c38e79bd9d813c521e9325: Status 404 returned error can't find the container with id 1c51e13f97f97b4bea36b3a2eca6b1e7c576edd1c7c38e79bd9d813c521e9325
Nov 03 16:23:14.285380 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:14.285329    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v" event={"ID":"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17","Type":"ContainerStarted","Data":"05ba2a48255e28ee4a90216d8521e94be693db2fa0e4d5335e8417b1670c6abe"}
Nov 03 16:23:14.285552 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:14.285399    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v" event={"ID":"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17","Type":"ContainerStarted","Data":"1c51e13f97f97b4bea36b3a2eca6b1e7c576edd1c7c38e79bd9d813c521e9325"}
Nov 03 16:23:14.317902 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:14.317844    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v" podStartSLOduration=1.317828593 podStartE2EDuration="1.317828593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:23:14.317269046 +0000 UTC m=+1208.330667219" watchObservedRunningTime="2026-11-03 16:23:14.317828593 +0000 UTC m=+1208.331226740"
Nov 03 16:23:14.329560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:14.329527    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-2prf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:23:14.329689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:14.329589    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" podUID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 16:23:14.363031 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:14.362998    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 16:23:14.363140 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:14.363052    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 16:23:15.297446 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:15.297379    6506 generic.go:356] "Generic (PLEG): container finished" podID="7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17" containerID="05ba2a48255e28ee4a90216d8521e94be693db2fa0e4d5335e8417b1670c6abe" exitCode=0
Nov 03 16:23:15.298008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:15.297470    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v" event={"ID":"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17","Type":"ContainerDied","Data":"05ba2a48255e28ee4a90216d8521e94be693db2fa0e4d5335e8417b1670c6abe"}
Nov 03 16:23:16.331166 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:16.331125    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-2prf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:23:16.331676 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:16.331193    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" podUID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:23:16.365641 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:16.365135    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:23:16.365641 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:16.365194    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:23:16.447069 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:16.447043    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"
Nov 03 16:23:16.581345 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:16.581245    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-kube-api-access-gjv5k\" (UniqueName: \"kubernetes.io/projected/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-kube-api-access-gjv5k\") pod \"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17\" (UID: \"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17\") "
Nov 03 16:23:16.581512 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:16.581345    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-maas-api-openshift-ca\") pod \"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17\" (UID: \"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17\") "
Nov 03 16:23:16.581512 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:16.581455    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-maas-api-xks-ca\") pod \"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17\" (UID: \"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17\") "
Nov 03 16:23:16.581847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:16.581814    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-maas-api-openshift-ca" pod "7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17" (UID: "7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:23:16.583562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:16.583541    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-maas-api-xks-ca" pod "7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17" (UID: "7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:23:16.583972 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:16.583955    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-kube-api-access-gjv5k" pod "7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17" (UID: "7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17"). InnerVolumeSpecName "kube-api-access-gjv5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:23:16.682382 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:16.682343    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:23:16.682382 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:16.682372    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjv5k\" (UniqueName: \"kubernetes.io/projected/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-kube-api-access-gjv5k\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:23:16.682382 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:16.682381    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:23:17.321845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:17.317418    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v" event={"ID":"7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17","Type":"ContainerDied","Data":"1c51e13f97f97b4bea36b3a2eca6b1e7c576edd1c7c38e79bd9d813c521e9325"}
Nov 03 16:23:17.321845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:17.317484    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c51e13f97f97b4bea36b3a2eca6b1e7c576edd1c7c38e79bd9d813c521e9325"
Nov 03 16:23:17.321845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:17.317479    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"
Nov 03 16:23:17.374065 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:17.374012    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-2prf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:23:17.374532 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:17.374135    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" podUID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:23:17.419819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:17.406978    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:23:17.419819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:17.407046    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:23:18.360305 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:18.360273    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:18.360824 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:18.360661    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:18.361825 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:18.361775    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:23:18.447977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:18.447942    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:23:18.448370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:18.448004    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:23:18.467984 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:18.467952    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:18.468519 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:18.468498    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:18.469194 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:18.469178    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:23:24.838222 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:24.838174    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:23:27.072471 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:27.072439    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:23:28.795789 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:23:28.795753    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:23:44.025475 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:44.025438    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"]
Nov 03 16:23:44.028175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:44.028149    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed386566-pr6jk"]
Nov 03 16:23:44.102340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:44.102301    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df4e02c-e4a4-417b-aec0-f08f98c10860" path="/var/lib/kubelet/pods/6df4e02c-e4a4-417b-aec0-f08f98c10860/volumes"
Nov 03 16:23:49.027614 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:49.027576    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"]
Nov 03 16:23:49.030864 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:49.030841    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d0cdbc6-6szhc"]
Nov 03 16:23:50.101895 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:50.101851    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746" path="/var/lib/kubelet/pods/a4c2e0c1-fbbb-4adf-bcae-07fc9ecb4746/volumes"
Nov 03 16:23:56.025357 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:56.025324    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"]
Nov 03 16:23:56.028501 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:56.028478    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41438547-lmlcx"]
Nov 03 16:23:56.102178 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:23:56.102148    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070f7662-d077-4fa3-bcf7-59ceb9eecb04" path="/var/lib/kubelet/pods/070f7662-d077-4fa3-bcf7-59ceb9eecb04/volumes"
Nov 03 16:23:58.795616 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:23:58.795568    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:24:02.028772 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:02.028734    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"]
Nov 03 16:24:02.032987 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:02.032955    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"]
Nov 03 16:24:02.035430 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:02.035404    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e857453-x2c4b"]
Nov 03 16:24:02.037361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:02.037341    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2956e28-8ccs9"]
Nov 03 16:24:02.102498 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:02.102467    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df53eb88-f023-4084-9e6a-eecbada27f7a" path="/var/lib/kubelet/pods/df53eb88-f023-4084-9e6a-eecbada27f7a/volumes"
Nov 03 16:24:02.102905 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:02.102892    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e94174aa-83ef-4037-861a-81dfbb6894d1" path="/var/lib/kubelet/pods/e94174aa-83ef-4037-861a-81dfbb6894d1/volumes"
Nov 03 16:24:04.405977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:04.405942    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-2prf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:24:04.406375 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:04.406028    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" podUID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:24:04.526820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:04.523563    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:24:04.526820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:04.523655    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:24:09.505291 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.505255    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"]
Nov 03 16:24:09.505679 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.505659    6506 cpu_manager.go:414] "containerMap: removing container" podUID="7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17" containerName="revoke-keys"
Nov 03 16:24:09.505679 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.505677    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17" containerName="revoke-keys"
Nov 03 16:24:09.505814 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.505785    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17" containerName="revoke-keys"
Nov 03 16:24:09.510061 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.510045    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"
Nov 03 16:24:09.517168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.516771    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:24:09.518004 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.517984    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:24:09.518117 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.518052    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:24:09.518117 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.518106    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:24:09.521959 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.521939    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"]
Nov 03 16:24:09.525436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.525411    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c033b89f-0694-4636-b45c-468c655c2e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-cq7cd\" (UID: \"c033b89f-0694-4636-b45c-468c655c2e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"
Nov 03 16:24:09.525518 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.525477    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c033b89f-0694-4636-b45c-468c655c2e3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-cq7cd\" (UID: \"c033b89f-0694-4636-b45c-468c655c2e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"
Nov 03 16:24:09.525609 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.525581    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c48q\" (UniqueName: \"kubernetes.io/projected/c033b89f-0694-4636-b45c-468c655c2e3c-kube-api-access-6c48q\") pod \"maas-api-revoke-sub-c1e271a9-cq7cd\" (UID: \"c033b89f-0694-4636-b45c-468c655c2e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"
Nov 03 16:24:09.629294 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.629257    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c033b89f-0694-4636-b45c-468c655c2e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-cq7cd\" (UID: \"c033b89f-0694-4636-b45c-468c655c2e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"
Nov 03 16:24:09.629561 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.629542    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c033b89f-0694-4636-b45c-468c655c2e3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-cq7cd\" (UID: \"c033b89f-0694-4636-b45c-468c655c2e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"
Nov 03 16:24:09.629706 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.629689    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c48q\" (UniqueName: \"kubernetes.io/projected/c033b89f-0694-4636-b45c-468c655c2e3c-kube-api-access-6c48q\") pod \"maas-api-revoke-sub-c1e271a9-cq7cd\" (UID: \"c033b89f-0694-4636-b45c-468c655c2e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"
Nov 03 16:24:09.631015 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.630991    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c033b89f-0694-4636-b45c-468c655c2e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-cq7cd\" (UID: \"c033b89f-0694-4636-b45c-468c655c2e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"
Nov 03 16:24:09.634138 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.634090    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c033b89f-0694-4636-b45c-468c655c2e3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-cq7cd\" (UID: \"c033b89f-0694-4636-b45c-468c655c2e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"
Nov 03 16:24:09.643296 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.643277    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c48q\" (UniqueName: \"kubernetes.io/projected/c033b89f-0694-4636-b45c-468c655c2e3c-kube-api-access-6c48q\") pod \"maas-api-revoke-sub-c1e271a9-cq7cd\" (UID: \"c033b89f-0694-4636-b45c-468c655c2e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"
Nov 03 16:24:09.825190 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.825151    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"
Nov 03 16:24:09.836398 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:09.836374    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:24:10.103050 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:10.103027    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"]
Nov 03 16:24:10.104784 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:24:10.104757    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc033b89f_0694_4636_b45c_468c655c2e3c.slice/crio-649ee15fdb7dbfe5fc8b6f823832e1a1b14a52aca8d3abde161137a25145f330 WatchSource:0}: Error finding container 649ee15fdb7dbfe5fc8b6f823832e1a1b14a52aca8d3abde161137a25145f330: Status 404 returned error can't find the container with id 649ee15fdb7dbfe5fc8b6f823832e1a1b14a52aca8d3abde161137a25145f330
Nov 03 16:24:10.643270 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:10.643227    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd" event={"ID":"c033b89f-0694-4636-b45c-468c655c2e3c","Type":"ContainerStarted","Data":"481af62c6cf326bdaca148480498960d325ae33783c6cbb4409d8f6b08dea912"}
Nov 03 16:24:10.643720 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:10.643287    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd" event={"ID":"c033b89f-0694-4636-b45c-468c655c2e3c","Type":"ContainerStarted","Data":"649ee15fdb7dbfe5fc8b6f823832e1a1b14a52aca8d3abde161137a25145f330"}
Nov 03 16:24:10.663949 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:10.663900    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd" podStartSLOduration=1.6638867080000002 podStartE2EDuration="1.663886708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:24:10.66306916 +0000 UTC m=+1264.676467306" watchObservedRunningTime="2026-11-03 16:24:10.663886708 +0000 UTC m=+1264.677284853"
Nov 03 16:24:11.034595 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:11.034557    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"]
Nov 03 16:24:11.036953 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:11.036926    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2051ba07-58d4t"]
Nov 03 16:24:11.650756 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:11.650720    6506 generic.go:356] "Generic (PLEG): container finished" podID="c033b89f-0694-4636-b45c-468c655c2e3c" containerID="481af62c6cf326bdaca148480498960d325ae33783c6cbb4409d8f6b08dea912" exitCode=0
Nov 03 16:24:11.651180 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:11.650777    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd" event={"ID":"c033b89f-0694-4636-b45c-468c655c2e3c","Type":"ContainerDied","Data":"481af62c6cf326bdaca148480498960d325ae33783c6cbb4409d8f6b08dea912"}
Nov 03 16:24:12.101538 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:12.101508    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd81e299-8d66-4599-b914-54674a52d4cb" path="/var/lib/kubelet/pods/fd81e299-8d66-4599-b914-54674a52d4cb/volumes"
Nov 03 16:24:12.787473 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:12.787450    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"
Nov 03 16:24:12.856026 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:12.855996    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c033b89f-0694-4636-b45c-468c655c2e3c-kube-api-access-6c48q\" (UniqueName: \"kubernetes.io/projected/c033b89f-0694-4636-b45c-468c655c2e3c-kube-api-access-6c48q\") pod \"c033b89f-0694-4636-b45c-468c655c2e3c\" (UID: \"c033b89f-0694-4636-b45c-468c655c2e3c\") "
Nov 03 16:24:12.856177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:12.856142    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c033b89f-0694-4636-b45c-468c655c2e3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c033b89f-0694-4636-b45c-468c655c2e3c-maas-api-openshift-ca\") pod \"c033b89f-0694-4636-b45c-468c655c2e3c\" (UID: \"c033b89f-0694-4636-b45c-468c655c2e3c\") "
Nov 03 16:24:12.856219 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:12.856179    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c033b89f-0694-4636-b45c-468c655c2e3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c033b89f-0694-4636-b45c-468c655c2e3c-maas-api-xks-ca\") pod \"c033b89f-0694-4636-b45c-468c655c2e3c\" (UID: \"c033b89f-0694-4636-b45c-468c655c2e3c\") "
Nov 03 16:24:12.856473 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:12.856441    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c033b89f-0694-4636-b45c-468c655c2e3c-maas-api-openshift-ca" pod "c033b89f-0694-4636-b45c-468c655c2e3c" (UID: "c033b89f-0694-4636-b45c-468c655c2e3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:24:12.858215 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:12.858188    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c033b89f-0694-4636-b45c-468c655c2e3c-kube-api-access-6c48q" pod "c033b89f-0694-4636-b45c-468c655c2e3c" (UID: "c033b89f-0694-4636-b45c-468c655c2e3c"). InnerVolumeSpecName "kube-api-access-6c48q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:24:12.858313 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:12.858276    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c033b89f-0694-4636-b45c-468c655c2e3c-maas-api-xks-ca" pod "c033b89f-0694-4636-b45c-468c655c2e3c" (UID: "c033b89f-0694-4636-b45c-468c655c2e3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:24:12.957480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:12.957424    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c033b89f-0694-4636-b45c-468c655c2e3c-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:24:12.957480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:12.957447    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c033b89f-0694-4636-b45c-468c655c2e3c-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:24:12.957480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:12.957458    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c48q\" (UniqueName: \"kubernetes.io/projected/c033b89f-0694-4636-b45c-468c655c2e3c-kube-api-access-6c48q\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:24:13.665178 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:13.665143    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd" event={"ID":"c033b89f-0694-4636-b45c-468c655c2e3c","Type":"ContainerDied","Data":"649ee15fdb7dbfe5fc8b6f823832e1a1b14a52aca8d3abde161137a25145f330"}
Nov 03 16:24:13.665178 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:13.665174    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"
Nov 03 16:24:13.665437 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:13.665193    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="649ee15fdb7dbfe5fc8b6f823832e1a1b14a52aca8d3abde161137a25145f330"
Nov 03 16:24:28.795272 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:24:28.795234    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:24:30.351977 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.351948    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"]
Nov 03 16:24:30.352366 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.352318    6506 cpu_manager.go:414] "containerMap: removing container" podUID="c033b89f-0694-4636-b45c-468c655c2e3c" containerName="revoke-keys"
Nov 03 16:24:30.352366 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.352328    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c033b89f-0694-4636-b45c-468c655c2e3c" containerName="revoke-keys"
Nov 03 16:24:30.352443 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.352409    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="c033b89f-0694-4636-b45c-468c655c2e3c" containerName="revoke-keys"
Nov 03 16:24:30.355555 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.355537    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"
Nov 03 16:24:30.363717 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.363693    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:24:30.363856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.363720    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:24:30.365447 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.365424    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"]
Nov 03 16:24:30.407156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.407131    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:24:30.407156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.407135    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:24:30.439770 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.439736    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58acba74-0ea8-41a2-8bc6-3c15c3257e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m6h4d\" (UID: \"58acba74-0ea8-41a2-8bc6-3c15c3257e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"
Nov 03 16:24:30.439967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.439815    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58acba74-0ea8-41a2-8bc6-3c15c3257e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m6h4d\" (UID: \"58acba74-0ea8-41a2-8bc6-3c15c3257e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"
Nov 03 16:24:30.439967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.439935    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5jkh\" (UniqueName: \"kubernetes.io/projected/58acba74-0ea8-41a2-8bc6-3c15c3257e21-kube-api-access-c5jkh\") pod \"maas-api-revoke-sub-2e1f5b5f-m6h4d\" (UID: \"58acba74-0ea8-41a2-8bc6-3c15c3257e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"
Nov 03 16:24:30.540667 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.540632    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58acba74-0ea8-41a2-8bc6-3c15c3257e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m6h4d\" (UID: \"58acba74-0ea8-41a2-8bc6-3c15c3257e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"
Nov 03 16:24:30.540866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.540688    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58acba74-0ea8-41a2-8bc6-3c15c3257e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m6h4d\" (UID: \"58acba74-0ea8-41a2-8bc6-3c15c3257e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"
Nov 03 16:24:30.540866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.540719    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5jkh\" (UniqueName: \"kubernetes.io/projected/58acba74-0ea8-41a2-8bc6-3c15c3257e21-kube-api-access-c5jkh\") pod \"maas-api-revoke-sub-2e1f5b5f-m6h4d\" (UID: \"58acba74-0ea8-41a2-8bc6-3c15c3257e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"
Nov 03 16:24:30.541391 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.541372    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58acba74-0ea8-41a2-8bc6-3c15c3257e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m6h4d\" (UID: \"58acba74-0ea8-41a2-8bc6-3c15c3257e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"
Nov 03 16:24:30.543146 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.543126    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58acba74-0ea8-41a2-8bc6-3c15c3257e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m6h4d\" (UID: \"58acba74-0ea8-41a2-8bc6-3c15c3257e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"
Nov 03 16:24:30.556457 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.556429    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5jkh\" (UniqueName: \"kubernetes.io/projected/58acba74-0ea8-41a2-8bc6-3c15c3257e21-kube-api-access-c5jkh\") pod \"maas-api-revoke-sub-2e1f5b5f-m6h4d\" (UID: \"58acba74-0ea8-41a2-8bc6-3c15c3257e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"
Nov 03 16:24:30.670508 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.670426    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"
Nov 03 16:24:30.953252 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:30.953221    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"]
Nov 03 16:24:30.954373 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:24:30.954347    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58acba74_0ea8_41a2_8bc6_3c15c3257e21.slice/crio-c393adb3bd53dc5b72135295e3b9dd654135d5c2571288d9128da6ea41bc6c4f WatchSource:0}: Error finding container c393adb3bd53dc5b72135295e3b9dd654135d5c2571288d9128da6ea41bc6c4f: Status 404 returned error can't find the container with id c393adb3bd53dc5b72135295e3b9dd654135d5c2571288d9128da6ea41bc6c4f
Nov 03 16:24:31.819241 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:31.819211    6506 generic.go:356] "Generic (PLEG): container finished" podID="58acba74-0ea8-41a2-8bc6-3c15c3257e21" containerID="b532841c533e68ae56b46e17bb56671161fb92621078814137199cdcfded1c9e" exitCode=0
Nov 03 16:24:31.819586 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:31.819257    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d" event={"ID":"58acba74-0ea8-41a2-8bc6-3c15c3257e21","Type":"ContainerDied","Data":"b532841c533e68ae56b46e17bb56671161fb92621078814137199cdcfded1c9e"}
Nov 03 16:24:31.819586 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:31.819319    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d" event={"ID":"58acba74-0ea8-41a2-8bc6-3c15c3257e21","Type":"ContainerStarted","Data":"c393adb3bd53dc5b72135295e3b9dd654135d5c2571288d9128da6ea41bc6c4f"}
Nov 03 16:24:32.972056 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:32.972026    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"
Nov 03 16:24:33.063862 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:33.063830    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58acba74-0ea8-41a2-8bc6-3c15c3257e21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58acba74-0ea8-41a2-8bc6-3c15c3257e21-maas-api-openshift-ca\") pod \"58acba74-0ea8-41a2-8bc6-3c15c3257e21\" (UID: \"58acba74-0ea8-41a2-8bc6-3c15c3257e21\") "
Nov 03 16:24:33.064021 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:33.063901    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58acba74-0ea8-41a2-8bc6-3c15c3257e21-kube-api-access-c5jkh\" (UniqueName: \"kubernetes.io/projected/58acba74-0ea8-41a2-8bc6-3c15c3257e21-kube-api-access-c5jkh\") pod \"58acba74-0ea8-41a2-8bc6-3c15c3257e21\" (UID: \"58acba74-0ea8-41a2-8bc6-3c15c3257e21\") "
Nov 03 16:24:33.064021 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:33.063980    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58acba74-0ea8-41a2-8bc6-3c15c3257e21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58acba74-0ea8-41a2-8bc6-3c15c3257e21-maas-api-xks-ca\") pod \"58acba74-0ea8-41a2-8bc6-3c15c3257e21\" (UID: \"58acba74-0ea8-41a2-8bc6-3c15c3257e21\") "
Nov 03 16:24:33.064230 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:33.064210    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58acba74-0ea8-41a2-8bc6-3c15c3257e21-maas-api-openshift-ca" pod "58acba74-0ea8-41a2-8bc6-3c15c3257e21" (UID: "58acba74-0ea8-41a2-8bc6-3c15c3257e21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:24:33.064367 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:33.064353    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58acba74-0ea8-41a2-8bc6-3c15c3257e21-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:24:33.066057 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:33.066034    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58acba74-0ea8-41a2-8bc6-3c15c3257e21-maas-api-xks-ca" pod "58acba74-0ea8-41a2-8bc6-3c15c3257e21" (UID: "58acba74-0ea8-41a2-8bc6-3c15c3257e21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:24:33.066153 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:33.066104    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58acba74-0ea8-41a2-8bc6-3c15c3257e21-kube-api-access-c5jkh" pod "58acba74-0ea8-41a2-8bc6-3c15c3257e21" (UID: "58acba74-0ea8-41a2-8bc6-3c15c3257e21"). InnerVolumeSpecName "kube-api-access-c5jkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:24:33.165541 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:33.165472    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5jkh\" (UniqueName: \"kubernetes.io/projected/58acba74-0ea8-41a2-8bc6-3c15c3257e21-kube-api-access-c5jkh\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:24:33.165541 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:33.165500    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58acba74-0ea8-41a2-8bc6-3c15c3257e21-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:24:33.831847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:33.831815    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"
Nov 03 16:24:33.831847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:33.831833    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d" event={"ID":"58acba74-0ea8-41a2-8bc6-3c15c3257e21","Type":"ContainerDied","Data":"c393adb3bd53dc5b72135295e3b9dd654135d5c2571288d9128da6ea41bc6c4f"}
Nov 03 16:24:33.832063 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:33.831911    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c393adb3bd53dc5b72135295e3b9dd654135d5c2571288d9128da6ea41bc6c4f"
Nov 03 16:24:41.354079 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.354039    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"]
Nov 03 16:24:41.354516 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.354403    6506 cpu_manager.go:414] "containerMap: removing container" podUID="58acba74-0ea8-41a2-8bc6-3c15c3257e21" containerName="revoke-keys"
Nov 03 16:24:41.354516 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.354416    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58acba74-0ea8-41a2-8bc6-3c15c3257e21" containerName="revoke-keys"
Nov 03 16:24:41.354516 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.354496    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="58acba74-0ea8-41a2-8bc6-3c15c3257e21" containerName="revoke-keys"
Nov 03 16:24:41.357873 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.357857    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"
Nov 03 16:24:41.366061 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.365809    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:24:41.366061 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.365821    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:24:41.366061 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.365913    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:24:41.366061 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.365915    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:24:41.367665 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.367643    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"]
Nov 03 16:24:41.448944 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.448916    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-qh9n8\" (UID: \"c472bb55-e437-4583-a52f-dc2ec9d8b4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"
Nov 03 16:24:41.449112 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.448970    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-qh9n8\" (UID: \"c472bb55-e437-4583-a52f-dc2ec9d8b4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"
Nov 03 16:24:41.449112 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.449059    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxttv\" (UniqueName: \"kubernetes.io/projected/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-kube-api-access-mxttv\") pod \"maas-api-revoke-sub-ec843018-qh9n8\" (UID: \"c472bb55-e437-4583-a52f-dc2ec9d8b4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"
Nov 03 16:24:41.549845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.549773    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-qh9n8\" (UID: \"c472bb55-e437-4583-a52f-dc2ec9d8b4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"
Nov 03 16:24:41.550025 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.549861    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-qh9n8\" (UID: \"c472bb55-e437-4583-a52f-dc2ec9d8b4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"
Nov 03 16:24:41.550025 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.549892    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxttv\" (UniqueName: \"kubernetes.io/projected/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-kube-api-access-mxttv\") pod \"maas-api-revoke-sub-ec843018-qh9n8\" (UID: \"c472bb55-e437-4583-a52f-dc2ec9d8b4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"
Nov 03 16:24:41.550531 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.550498    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-qh9n8\" (UID: \"c472bb55-e437-4583-a52f-dc2ec9d8b4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"
Nov 03 16:24:41.552235 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.552214    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-qh9n8\" (UID: \"c472bb55-e437-4583-a52f-dc2ec9d8b4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"
Nov 03 16:24:41.562272 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.562248    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxttv\" (UniqueName: \"kubernetes.io/projected/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-kube-api-access-mxttv\") pod \"maas-api-revoke-sub-ec843018-qh9n8\" (UID: \"c472bb55-e437-4583-a52f-dc2ec9d8b4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"
Nov 03 16:24:41.672506 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.672424    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"
Nov 03 16:24:41.958683 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:41.958655    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"]
Nov 03 16:24:41.959460 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:24:41.959432    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc472bb55_e437_4583_a52f_dc2ec9d8b4ca.slice/crio-2e1189c8389abb8b4ea19303894ccf45bab913121330d8a94dd8b32dfc2bf27c WatchSource:0}: Error finding container 2e1189c8389abb8b4ea19303894ccf45bab913121330d8a94dd8b32dfc2bf27c: Status 404 returned error can't find the container with id 2e1189c8389abb8b4ea19303894ccf45bab913121330d8a94dd8b32dfc2bf27c
Nov 03 16:24:42.886820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:42.886780    6506 generic.go:356] "Generic (PLEG): container finished" podID="c472bb55-e437-4583-a52f-dc2ec9d8b4ca" containerID="4d2727a685d1da6b585950fd7c0b90eaef79d95b09240e1d4267250ccd2ed60e" exitCode=0
Nov 03 16:24:42.887180 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:42.886865    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8" event={"ID":"c472bb55-e437-4583-a52f-dc2ec9d8b4ca","Type":"ContainerDied","Data":"4d2727a685d1da6b585950fd7c0b90eaef79d95b09240e1d4267250ccd2ed60e"}
Nov 03 16:24:42.887180 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:42.886912    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8" event={"ID":"c472bb55-e437-4583-a52f-dc2ec9d8b4ca","Type":"ContainerStarted","Data":"2e1189c8389abb8b4ea19303894ccf45bab913121330d8a94dd8b32dfc2bf27c"}
Nov 03 16:24:44.064996 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:44.064968    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"
Nov 03 16:24:44.180586 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:44.180490    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-kube-api-access-mxttv\" (UniqueName: \"kubernetes.io/projected/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-kube-api-access-mxttv\") pod \"c472bb55-e437-4583-a52f-dc2ec9d8b4ca\" (UID: \"c472bb55-e437-4583-a52f-dc2ec9d8b4ca\") "
Nov 03 16:24:44.180758 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:44.180606    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-maas-api-xks-ca\") pod \"c472bb55-e437-4583-a52f-dc2ec9d8b4ca\" (UID: \"c472bb55-e437-4583-a52f-dc2ec9d8b4ca\") "
Nov 03 16:24:44.180758 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:44.180638    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-maas-api-openshift-ca\") pod \"c472bb55-e437-4583-a52f-dc2ec9d8b4ca\" (UID: \"c472bb55-e437-4583-a52f-dc2ec9d8b4ca\") "
Nov 03 16:24:44.181144 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:44.181104    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-maas-api-openshift-ca" pod "c472bb55-e437-4583-a52f-dc2ec9d8b4ca" (UID: "c472bb55-e437-4583-a52f-dc2ec9d8b4ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:24:44.183373 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:44.183345    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-maas-api-xks-ca" pod "c472bb55-e437-4583-a52f-dc2ec9d8b4ca" (UID: "c472bb55-e437-4583-a52f-dc2ec9d8b4ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:24:44.183466 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:44.183373    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-kube-api-access-mxttv" pod "c472bb55-e437-4583-a52f-dc2ec9d8b4ca" (UID: "c472bb55-e437-4583-a52f-dc2ec9d8b4ca"). InnerVolumeSpecName "kube-api-access-mxttv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:24:44.281592 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:44.281558    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxttv\" (UniqueName: \"kubernetes.io/projected/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-kube-api-access-mxttv\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:24:44.281592 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:44.281586    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:24:44.281592 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:44.281595    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c472bb55-e437-4583-a52f-dc2ec9d8b4ca-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:24:44.903763 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:44.903734    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"
Nov 03 16:24:44.903963 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:44.903777    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8" event={"ID":"c472bb55-e437-4583-a52f-dc2ec9d8b4ca","Type":"ContainerDied","Data":"2e1189c8389abb8b4ea19303894ccf45bab913121330d8a94dd8b32dfc2bf27c"}
Nov 03 16:24:44.903963 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:44.903845    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e1189c8389abb8b4ea19303894ccf45bab913121330d8a94dd8b32dfc2bf27c"
Nov 03 16:24:45.151820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:45.151237    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:24:46.136971 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:46.136934    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:24:46.137291 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:24:46.137258    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" podUID="13e172cb-f664-4da0-a922-08b5a0eb29d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:24:58.795578 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:24:58.795534    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:25:06.465519 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:06.465488    6506 scope.go:122] "RemoveContainer" containerID="19e4a6e549cc3576e37e369388b3d7542e2b46a4438b3bea0a6a7d3f5abffd1f"
Nov 03 16:25:06.474835 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:06.474813    6506 scope.go:122] "RemoveContainer" containerID="be04111f0641df101b59b73a2add9c4e5c6d7d00f03a11de8c3e8cd4e1f2a247"
Nov 03 16:25:06.483110 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:06.483083    6506 scope.go:122] "RemoveContainer" containerID="88c47481e33da655eebc0d4edc6263cb29ee03de259739dc94e7e9639e8e3035"
Nov 03 16:25:06.491531 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:06.491514    6506 scope.go:122] "RemoveContainer" containerID="48e0b5c997d48f979e428bb94d9c148140c082fee844f8727e1533b026cb3ffb"
Nov 03 16:25:06.499722 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:06.499707    6506 scope.go:122] "RemoveContainer" containerID="6c6a7dc8c0221371cf7d33a6ec16b8640d739728d23df5185f135b72f1e4b68e"
Nov 03 16:25:07.540719 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.540660    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"]
Nov 03 16:25:07.541180 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.541094    6506 cpu_manager.go:414] "containerMap: removing container" podUID="c472bb55-e437-4583-a52f-dc2ec9d8b4ca" containerName="revoke-keys"
Nov 03 16:25:07.541180 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.541107    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c472bb55-e437-4583-a52f-dc2ec9d8b4ca" containerName="revoke-keys"
Nov 03 16:25:07.541308 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.541202    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="c472bb55-e437-4583-a52f-dc2ec9d8b4ca" containerName="revoke-keys"
Nov 03 16:25:07.546099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.546083    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"
Nov 03 16:25:07.552635 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.552123    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:25:07.552635 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.552157    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:25:07.552635 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.552310    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:25:07.554844 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.552121    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:25:07.555602 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.555584    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"]
Nov 03 16:25:07.642208 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.642179    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhw5c\" (UniqueName: \"kubernetes.io/projected/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-kube-api-access-qhw5c\") pod \"maas-api-revoke-sub-046e1343-qd277\" (UID: \"248fadfb-54a8-4b6c-a13b-9c99b2a25b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"
Nov 03 16:25:07.642360 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.642217    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-qd277\" (UID: \"248fadfb-54a8-4b6c-a13b-9c99b2a25b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"
Nov 03 16:25:07.642360 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.642258    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-qd277\" (UID: \"248fadfb-54a8-4b6c-a13b-9c99b2a25b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"
Nov 03 16:25:07.743693 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.743659    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhw5c\" (UniqueName: \"kubernetes.io/projected/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-kube-api-access-qhw5c\") pod \"maas-api-revoke-sub-046e1343-qd277\" (UID: \"248fadfb-54a8-4b6c-a13b-9c99b2a25b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"
Nov 03 16:25:07.743869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.743704    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-qd277\" (UID: \"248fadfb-54a8-4b6c-a13b-9c99b2a25b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"
Nov 03 16:25:07.743869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.743724    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-qd277\" (UID: \"248fadfb-54a8-4b6c-a13b-9c99b2a25b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"
Nov 03 16:25:07.744394 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.744376    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-qd277\" (UID: \"248fadfb-54a8-4b6c-a13b-9c99b2a25b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"
Nov 03 16:25:07.746147 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.746126    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-qd277\" (UID: \"248fadfb-54a8-4b6c-a13b-9c99b2a25b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"
Nov 03 16:25:07.756126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.756102    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhw5c\" (UniqueName: \"kubernetes.io/projected/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-kube-api-access-qhw5c\") pod \"maas-api-revoke-sub-046e1343-qd277\" (UID: \"248fadfb-54a8-4b6c-a13b-9c99b2a25b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"
Nov 03 16:25:07.861015 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:07.860938    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"
Nov 03 16:25:08.138812 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:08.138773    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"]
Nov 03 16:25:08.139597 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:25:08.139570    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248fadfb_54a8_4b6c_a13b_9c99b2a25b74.slice/crio-f068f95f61e7ec2ab9f45a23421091f70b48aa935784376e50d53ee4eee4a162 WatchSource:0}: Error finding container f068f95f61e7ec2ab9f45a23421091f70b48aa935784376e50d53ee4eee4a162: Status 404 returned error can't find the container with id f068f95f61e7ec2ab9f45a23421091f70b48aa935784376e50d53ee4eee4a162
Nov 03 16:25:09.071204 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:09.071169    6506 generic.go:356] "Generic (PLEG): container finished" podID="248fadfb-54a8-4b6c-a13b-9c99b2a25b74" containerID="17a760adf53390bcf4b0886db7dda72feccdaccba04dfcb7b826a235931f94fa" exitCode=0
Nov 03 16:25:09.071703 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:09.071261    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277" event={"ID":"248fadfb-54a8-4b6c-a13b-9c99b2a25b74","Type":"ContainerDied","Data":"17a760adf53390bcf4b0886db7dda72feccdaccba04dfcb7b826a235931f94fa"}
Nov 03 16:25:09.071703 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:09.071328    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277" event={"ID":"248fadfb-54a8-4b6c-a13b-9c99b2a25b74","Type":"ContainerStarted","Data":"f068f95f61e7ec2ab9f45a23421091f70b48aa935784376e50d53ee4eee4a162"}
Nov 03 16:25:09.885341 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:09.885304    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:25:10.212764 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:10.212742    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"
Nov 03 16:25:10.369670 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:10.369635    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-maas-api-openshift-ca\") pod \"248fadfb-54a8-4b6c-a13b-9c99b2a25b74\" (UID: \"248fadfb-54a8-4b6c-a13b-9c99b2a25b74\") "
Nov 03 16:25:10.369891 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:10.369696    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-kube-api-access-qhw5c\" (UniqueName: \"kubernetes.io/projected/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-kube-api-access-qhw5c\") pod \"248fadfb-54a8-4b6c-a13b-9c99b2a25b74\" (UID: \"248fadfb-54a8-4b6c-a13b-9c99b2a25b74\") "
Nov 03 16:25:10.369891 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:10.369777    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-maas-api-xks-ca\") pod \"248fadfb-54a8-4b6c-a13b-9c99b2a25b74\" (UID: \"248fadfb-54a8-4b6c-a13b-9c99b2a25b74\") "
Nov 03 16:25:10.370201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:10.370061    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-maas-api-openshift-ca" pod "248fadfb-54a8-4b6c-a13b-9c99b2a25b74" (UID: "248fadfb-54a8-4b6c-a13b-9c99b2a25b74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:25:10.371925 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:10.371907    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-maas-api-xks-ca" pod "248fadfb-54a8-4b6c-a13b-9c99b2a25b74" (UID: "248fadfb-54a8-4b6c-a13b-9c99b2a25b74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:25:10.372029 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:10.372007    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-kube-api-access-qhw5c" pod "248fadfb-54a8-4b6c-a13b-9c99b2a25b74" (UID: "248fadfb-54a8-4b6c-a13b-9c99b2a25b74"). InnerVolumeSpecName "kube-api-access-qhw5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:25:10.471480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:10.471402    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhw5c\" (UniqueName: \"kubernetes.io/projected/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-kube-api-access-qhw5c\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:25:10.471480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:10.471431    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:25:10.471480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:10.471441    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248fadfb-54a8-4b6c-a13b-9c99b2a25b74-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:25:11.084162 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:11.084135    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"
Nov 03 16:25:11.084332 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:11.084130    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277" event={"ID":"248fadfb-54a8-4b6c-a13b-9c99b2a25b74","Type":"ContainerDied","Data":"f068f95f61e7ec2ab9f45a23421091f70b48aa935784376e50d53ee4eee4a162"}
Nov 03 16:25:11.084332 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:11.084266    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f068f95f61e7ec2ab9f45a23421091f70b48aa935784376e50d53ee4eee4a162"
Nov 03 16:25:17.757383 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.757317    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"]
Nov 03 16:25:17.757852 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.757835    6506 cpu_manager.go:414] "containerMap: removing container" podUID="248fadfb-54a8-4b6c-a13b-9c99b2a25b74" containerName="revoke-keys"
Nov 03 16:25:17.757903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.757854    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248fadfb-54a8-4b6c-a13b-9c99b2a25b74" containerName="revoke-keys"
Nov 03 16:25:17.757943 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.757917    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="248fadfb-54a8-4b6c-a13b-9c99b2a25b74" containerName="revoke-keys"
Nov 03 16:25:17.761425 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.761405    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"
Nov 03 16:25:17.769588 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.769563    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:25:17.769736 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.769718    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:25:17.769942 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.769922    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:25:17.770075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.769940    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:25:17.774251 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.774229    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"]
Nov 03 16:25:17.838615 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.838576    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c39a05-9d40-489a-b0ee-c65f394ad8ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5r6m6\" (UID: \"48c39a05-9d40-489a-b0ee-c65f394ad8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"
Nov 03 16:25:17.838785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.838686    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4gc7\" (UniqueName: \"kubernetes.io/projected/48c39a05-9d40-489a-b0ee-c65f394ad8ff-kube-api-access-z4gc7\") pod \"maas-api-revoke-sub-d3fba056-5r6m6\" (UID: \"48c39a05-9d40-489a-b0ee-c65f394ad8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"
Nov 03 16:25:17.838785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.838733    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c39a05-9d40-489a-b0ee-c65f394ad8ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5r6m6\" (UID: \"48c39a05-9d40-489a-b0ee-c65f394ad8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"
Nov 03 16:25:17.939425 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.939392    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c39a05-9d40-489a-b0ee-c65f394ad8ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5r6m6\" (UID: \"48c39a05-9d40-489a-b0ee-c65f394ad8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"
Nov 03 16:25:17.939425 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.939430    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4gc7\" (UniqueName: \"kubernetes.io/projected/48c39a05-9d40-489a-b0ee-c65f394ad8ff-kube-api-access-z4gc7\") pod \"maas-api-revoke-sub-d3fba056-5r6m6\" (UID: \"48c39a05-9d40-489a-b0ee-c65f394ad8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"
Nov 03 16:25:17.939609 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.939467    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c39a05-9d40-489a-b0ee-c65f394ad8ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5r6m6\" (UID: \"48c39a05-9d40-489a-b0ee-c65f394ad8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"
Nov 03 16:25:17.940152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.940124    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c39a05-9d40-489a-b0ee-c65f394ad8ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5r6m6\" (UID: \"48c39a05-9d40-489a-b0ee-c65f394ad8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"
Nov 03 16:25:17.942013 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.941992    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c39a05-9d40-489a-b0ee-c65f394ad8ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5r6m6\" (UID: \"48c39a05-9d40-489a-b0ee-c65f394ad8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"
Nov 03 16:25:17.953177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:17.953152    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4gc7\" (UniqueName: \"kubernetes.io/projected/48c39a05-9d40-489a-b0ee-c65f394ad8ff-kube-api-access-z4gc7\") pod \"maas-api-revoke-sub-d3fba056-5r6m6\" (UID: \"48c39a05-9d40-489a-b0ee-c65f394ad8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"
Nov 03 16:25:18.076871 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:18.076838    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"
Nov 03 16:25:18.357590 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:18.357555    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"]
Nov 03 16:25:19.140883 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:19.140850    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6" event={"ID":"48c39a05-9d40-489a-b0ee-c65f394ad8ff","Type":"ContainerStarted","Data":"2e4d7a34be0f22969545404db89f94cb0fbd0073cc3220c549c767ee7e719b7a"}
Nov 03 16:25:19.141288 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:19.140907    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6" event={"ID":"48c39a05-9d40-489a-b0ee-c65f394ad8ff","Type":"ContainerStarted","Data":"72b76b7e63298a8a6646a057085e34886ad2b9e5bd9557e263e62edc36032de1"}
Nov 03 16:25:19.168032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:19.167971    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6" podStartSLOduration=2.167952099 podStartE2EDuration="2.167952099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:25:19.163341317 +0000 UTC m=+1333.176739464" watchObservedRunningTime="2026-11-03 16:25:19.167952099 +0000 UTC m=+1333.181350246"
Nov 03 16:25:20.166946 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:20.166907    6506 generic.go:356] "Generic (PLEG): container finished" podID="48c39a05-9d40-489a-b0ee-c65f394ad8ff" containerID="2e4d7a34be0f22969545404db89f94cb0fbd0073cc3220c549c767ee7e719b7a" exitCode=0
Nov 03 16:25:20.167669 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:20.167640    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6" event={"ID":"48c39a05-9d40-489a-b0ee-c65f394ad8ff","Type":"ContainerDied","Data":"2e4d7a34be0f22969545404db89f94cb0fbd0073cc3220c549c767ee7e719b7a"}
Nov 03 16:25:21.360242 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:21.360219    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"
Nov 03 16:25:21.402661 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:21.402630    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c39a05-9d40-489a-b0ee-c65f394ad8ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c39a05-9d40-489a-b0ee-c65f394ad8ff-maas-api-openshift-ca\") pod \"48c39a05-9d40-489a-b0ee-c65f394ad8ff\" (UID: \"48c39a05-9d40-489a-b0ee-c65f394ad8ff\") "
Nov 03 16:25:21.402865 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:21.402712    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48c39a05-9d40-489a-b0ee-c65f394ad8ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c39a05-9d40-489a-b0ee-c65f394ad8ff-maas-api-xks-ca\") pod \"48c39a05-9d40-489a-b0ee-c65f394ad8ff\" (UID: \"48c39a05-9d40-489a-b0ee-c65f394ad8ff\") "
Nov 03 16:25:21.402865 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:21.402782    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c39a05-9d40-489a-b0ee-c65f394ad8ff-kube-api-access-z4gc7\" (UniqueName: \"kubernetes.io/projected/48c39a05-9d40-489a-b0ee-c65f394ad8ff-kube-api-access-z4gc7\") pod \"48c39a05-9d40-489a-b0ee-c65f394ad8ff\" (UID: \"48c39a05-9d40-489a-b0ee-c65f394ad8ff\") "
Nov 03 16:25:21.403170 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:21.403126    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c39a05-9d40-489a-b0ee-c65f394ad8ff-maas-api-openshift-ca" pod "48c39a05-9d40-489a-b0ee-c65f394ad8ff" (UID: "48c39a05-9d40-489a-b0ee-c65f394ad8ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:25:21.408002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:21.407977    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c39a05-9d40-489a-b0ee-c65f394ad8ff-maas-api-xks-ca" pod "48c39a05-9d40-489a-b0ee-c65f394ad8ff" (UID: "48c39a05-9d40-489a-b0ee-c65f394ad8ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:25:21.408173 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:21.407980    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c39a05-9d40-489a-b0ee-c65f394ad8ff-kube-api-access-z4gc7" pod "48c39a05-9d40-489a-b0ee-c65f394ad8ff" (UID: "48c39a05-9d40-489a-b0ee-c65f394ad8ff"). InnerVolumeSpecName "kube-api-access-z4gc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:25:21.503572 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:21.503498    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c39a05-9d40-489a-b0ee-c65f394ad8ff-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:25:21.503572 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:21.503525    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c39a05-9d40-489a-b0ee-c65f394ad8ff-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:25:21.503572 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:21.503534    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4gc7\" (UniqueName: \"kubernetes.io/projected/48c39a05-9d40-489a-b0ee-c65f394ad8ff-kube-api-access-z4gc7\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:25:22.184301 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:22.184274    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"
Nov 03 16:25:22.184452 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:22.184310    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6" event={"ID":"48c39a05-9d40-489a-b0ee-c65f394ad8ff","Type":"ContainerDied","Data":"72b76b7e63298a8a6646a057085e34886ad2b9e5bd9557e263e62edc36032de1"}
Nov 03 16:25:22.184452 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:22.184361    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72b76b7e63298a8a6646a057085e34886ad2b9e5bd9557e263e62edc36032de1"
Nov 03 16:25:23.570185 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:23.570147    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:25:28.795358 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:25:28.795324    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:25:38.511959 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.511922    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"]
Nov 03 16:25:38.512365 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.512285    6506 cpu_manager.go:414] "containerMap: removing container" podUID="48c39a05-9d40-489a-b0ee-c65f394ad8ff" containerName="revoke-keys"
Nov 03 16:25:38.512365 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.512295    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c39a05-9d40-489a-b0ee-c65f394ad8ff" containerName="revoke-keys"
Nov 03 16:25:38.512440 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.512376    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c39a05-9d40-489a-b0ee-c65f394ad8ff" containerName="revoke-keys"
Nov 03 16:25:38.515663 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.515643    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"
Nov 03 16:25:38.526025 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.526004    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:25:38.526112 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.526040    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:25:38.526443 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.526425    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:25:38.526530 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.526455    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:25:38.529759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.529737    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"]
Nov 03 16:25:38.572689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.572663    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsm76\" (UniqueName: \"kubernetes.io/projected/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-kube-api-access-hsm76\") pod \"maas-api-revoke-sub-ae594979-xzb9p\" (UID: \"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"
Nov 03 16:25:38.572852 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.572700    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-xzb9p\" (UID: \"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"
Nov 03 16:25:38.572852 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.572716    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-xzb9p\" (UID: \"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"
Nov 03 16:25:38.674129 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.674096    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsm76\" (UniqueName: \"kubernetes.io/projected/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-kube-api-access-hsm76\") pod \"maas-api-revoke-sub-ae594979-xzb9p\" (UID: \"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"
Nov 03 16:25:38.674272 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.674134    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-xzb9p\" (UID: \"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"
Nov 03 16:25:38.674272 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.674150    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-xzb9p\" (UID: \"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"
Nov 03 16:25:38.674883 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.674865    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-xzb9p\" (UID: \"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"
Nov 03 16:25:38.676697 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.676681    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-xzb9p\" (UID: \"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"
Nov 03 16:25:38.687347 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.687328    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsm76\" (UniqueName: \"kubernetes.io/projected/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-kube-api-access-hsm76\") pod \"maas-api-revoke-sub-ae594979-xzb9p\" (UID: \"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"
Nov 03 16:25:38.828157 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:38.828131    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"
Nov 03 16:25:39.104925 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:39.104895    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"]
Nov 03 16:25:39.105145 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:25:39.105116    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b5fa86_c589_429f_bb0f_2a5f2c7ec248.slice/crio-adea51945d38c100bfc44c41521fa60e7c4612f3d03930322321d7c7c8d82c72 WatchSource:0}: Error finding container adea51945d38c100bfc44c41521fa60e7c4612f3d03930322321d7c7c8d82c72: Status 404 returned error can't find the container with id adea51945d38c100bfc44c41521fa60e7c4612f3d03930322321d7c7c8d82c72
Nov 03 16:25:39.314066 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:39.314017    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p" event={"ID":"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248","Type":"ContainerStarted","Data":"adea51945d38c100bfc44c41521fa60e7c4612f3d03930322321d7c7c8d82c72"}
Nov 03 16:25:40.321671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:40.321640    6506 generic.go:356] "Generic (PLEG): container finished" podID="d7b5fa86-c589-429f-bb0f-2a5f2c7ec248" containerID="425216545d382bc9f5e92b23e86f48f6d8e5137f8c3402d022c9ef2a21236d98" exitCode=0
Nov 03 16:25:40.322126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:40.321735    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p" event={"ID":"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248","Type":"ContainerDied","Data":"425216545d382bc9f5e92b23e86f48f6d8e5137f8c3402d022c9ef2a21236d98"}
Nov 03 16:25:41.461095 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:41.461070    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"
Nov 03 16:25:41.601685 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:41.601606    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-kube-api-access-hsm76\" (UniqueName: \"kubernetes.io/projected/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-kube-api-access-hsm76\") pod \"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248\" (UID: \"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248\") "
Nov 03 16:25:41.601685 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:41.601644    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-maas-api-openshift-ca\") pod \"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248\" (UID: \"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248\") "
Nov 03 16:25:41.601685 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:41.601679    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-maas-api-xks-ca\") pod \"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248\" (UID: \"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248\") "
Nov 03 16:25:41.602072 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:41.602044    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-maas-api-openshift-ca" pod "d7b5fa86-c589-429f-bb0f-2a5f2c7ec248" (UID: "d7b5fa86-c589-429f-bb0f-2a5f2c7ec248"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:25:41.603906 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:41.603877    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-maas-api-xks-ca" pod "d7b5fa86-c589-429f-bb0f-2a5f2c7ec248" (UID: "d7b5fa86-c589-429f-bb0f-2a5f2c7ec248"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:25:41.603906 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:41.603886    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-kube-api-access-hsm76" pod "d7b5fa86-c589-429f-bb0f-2a5f2c7ec248" (UID: "d7b5fa86-c589-429f-bb0f-2a5f2c7ec248"). InnerVolumeSpecName "kube-api-access-hsm76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:25:41.703246 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:41.703210    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsm76\" (UniqueName: \"kubernetes.io/projected/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-kube-api-access-hsm76\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:25:41.703246 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:41.703241    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:25:41.703246 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:41.703251    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:25:42.334605 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:42.334579    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"
Nov 03 16:25:42.334773 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:42.334620    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p" event={"ID":"d7b5fa86-c589-429f-bb0f-2a5f2c7ec248","Type":"ContainerDied","Data":"adea51945d38c100bfc44c41521fa60e7c4612f3d03930322321d7c7c8d82c72"}
Nov 03 16:25:42.334773 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:42.334669    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adea51945d38c100bfc44c41521fa60e7c4612f3d03930322321d7c7c8d82c72"
Nov 03 16:25:43.025966 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:43.025934    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:25:49.031168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:49.031096    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"]
Nov 03 16:25:49.033467 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:49.033445    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7503e5f-lzvjk"]
Nov 03 16:25:50.101759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:50.101727    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5f4ac5-141c-4b00-869d-942cc8198b1d" path="/var/lib/kubelet/pods/ca5f4ac5-141c-4b00-869d-942cc8198b1d/volumes"
Nov 03 16:25:53.030927 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:53.030892    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"]
Nov 03 16:25:53.090707 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:53.090655    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d7d6c3e-wxxjc"]
Nov 03 16:25:54.101967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:54.101935    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae685f25-0167-4586-ba50-bf37f08ed5e6" path="/var/lib/kubelet/pods/ae685f25-0167-4586-ba50-bf37f08ed5e6/volumes"
Nov 03 16:25:56.924664 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:56.924626    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"]
Nov 03 16:25:56.925092 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:56.925022    6506 cpu_manager.go:414] "containerMap: removing container" podUID="d7b5fa86-c589-429f-bb0f-2a5f2c7ec248" containerName="revoke-keys"
Nov 03 16:25:56.925092 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:56.925033    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b5fa86-c589-429f-bb0f-2a5f2c7ec248" containerName="revoke-keys"
Nov 03 16:25:56.925176 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:56.925112    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b5fa86-c589-429f-bb0f-2a5f2c7ec248" containerName="revoke-keys"
Nov 03 16:25:56.928352 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:56.928336    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"
Nov 03 16:25:56.933887 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:56.933862    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:25:56.934104 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:56.934088    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:25:56.934314 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:56.934164    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:25:56.934570 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:56.934518    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:25:56.936624 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:56.936598    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"]
Nov 03 16:25:57.018013 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:57.017981    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11a05bb-37ae-45ba-b257-7553097de947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2rkbb\" (UID: \"c11a05bb-37ae-45ba-b257-7553097de947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"
Nov 03 16:25:57.018013 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:57.018018    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11a05bb-37ae-45ba-b257-7553097de947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2rkbb\" (UID: \"c11a05bb-37ae-45ba-b257-7553097de947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"
Nov 03 16:25:57.018213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:57.018181    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hfz8\" (UniqueName: \"kubernetes.io/projected/c11a05bb-37ae-45ba-b257-7553097de947-kube-api-access-2hfz8\") pod \"maas-api-revoke-sub-a4aef7d7-2rkbb\" (UID: \"c11a05bb-37ae-45ba-b257-7553097de947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"
Nov 03 16:25:57.119148 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:57.119113    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hfz8\" (UniqueName: \"kubernetes.io/projected/c11a05bb-37ae-45ba-b257-7553097de947-kube-api-access-2hfz8\") pod \"maas-api-revoke-sub-a4aef7d7-2rkbb\" (UID: \"c11a05bb-37ae-45ba-b257-7553097de947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"
Nov 03 16:25:57.119325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:57.119174    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11a05bb-37ae-45ba-b257-7553097de947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2rkbb\" (UID: \"c11a05bb-37ae-45ba-b257-7553097de947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"
Nov 03 16:25:57.119325 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:57.119193    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11a05bb-37ae-45ba-b257-7553097de947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2rkbb\" (UID: \"c11a05bb-37ae-45ba-b257-7553097de947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"
Nov 03 16:25:57.119809 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:57.119769    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11a05bb-37ae-45ba-b257-7553097de947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2rkbb\" (UID: \"c11a05bb-37ae-45ba-b257-7553097de947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"
Nov 03 16:25:57.121619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:57.121601    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11a05bb-37ae-45ba-b257-7553097de947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2rkbb\" (UID: \"c11a05bb-37ae-45ba-b257-7553097de947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"
Nov 03 16:25:57.142433 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:57.142408    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hfz8\" (UniqueName: \"kubernetes.io/projected/c11a05bb-37ae-45ba-b257-7553097de947-kube-api-access-2hfz8\") pod \"maas-api-revoke-sub-a4aef7d7-2rkbb\" (UID: \"c11a05bb-37ae-45ba-b257-7553097de947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"
Nov 03 16:25:57.242166 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:57.242071    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"
Nov 03 16:25:57.532847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:57.532819    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"]
Nov 03 16:25:57.533466 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:25:57.533435    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11a05bb_37ae_45ba_b257_7553097de947.slice/crio-170ffa70ad099fd747bb03d4bedcb344c650dc15073e06678f437c08ab2ff557 WatchSource:0}: Error finding container 170ffa70ad099fd747bb03d4bedcb344c650dc15073e06678f437c08ab2ff557: Status 404 returned error can't find the container with id 170ffa70ad099fd747bb03d4bedcb344c650dc15073e06678f437c08ab2ff557
Nov 03 16:25:58.430135 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:58.430097    6506 generic.go:356] "Generic (PLEG): container finished" podID="c11a05bb-37ae-45ba-b257-7553097de947" containerID="b53515f28e07966dc6d67a7df38657f0743cb08fff127f3ce1ab267c6fb87adc" exitCode=0
Nov 03 16:25:58.430518 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:58.430190    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb" event={"ID":"c11a05bb-37ae-45ba-b257-7553097de947","Type":"ContainerDied","Data":"b53515f28e07966dc6d67a7df38657f0743cb08fff127f3ce1ab267c6fb87adc"}
Nov 03 16:25:58.430518 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:58.430253    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb" event={"ID":"c11a05bb-37ae-45ba-b257-7553097de947","Type":"ContainerStarted","Data":"170ffa70ad099fd747bb03d4bedcb344c650dc15073e06678f437c08ab2ff557"}
Nov 03 16:25:58.796265 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:25:58.796227    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:25:59.567164 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:59.567141    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"
Nov 03 16:25:59.643624 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:59.643593    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c11a05bb-37ae-45ba-b257-7553097de947-kube-api-access-2hfz8\" (UniqueName: \"kubernetes.io/projected/c11a05bb-37ae-45ba-b257-7553097de947-kube-api-access-2hfz8\") pod \"c11a05bb-37ae-45ba-b257-7553097de947\" (UID: \"c11a05bb-37ae-45ba-b257-7553097de947\") "
Nov 03 16:25:59.643772 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:59.643650    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c11a05bb-37ae-45ba-b257-7553097de947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11a05bb-37ae-45ba-b257-7553097de947-maas-api-openshift-ca\") pod \"c11a05bb-37ae-45ba-b257-7553097de947\" (UID: \"c11a05bb-37ae-45ba-b257-7553097de947\") "
Nov 03 16:25:59.643856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:59.643766    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c11a05bb-37ae-45ba-b257-7553097de947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11a05bb-37ae-45ba-b257-7553097de947-maas-api-xks-ca\") pod \"c11a05bb-37ae-45ba-b257-7553097de947\" (UID: \"c11a05bb-37ae-45ba-b257-7553097de947\") "
Nov 03 16:25:59.644041 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:59.644018    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c11a05bb-37ae-45ba-b257-7553097de947-maas-api-openshift-ca" pod "c11a05bb-37ae-45ba-b257-7553097de947" (UID: "c11a05bb-37ae-45ba-b257-7553097de947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:25:59.645894 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:59.645868    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11a05bb-37ae-45ba-b257-7553097de947-kube-api-access-2hfz8" pod "c11a05bb-37ae-45ba-b257-7553097de947" (UID: "c11a05bb-37ae-45ba-b257-7553097de947"). InnerVolumeSpecName "kube-api-access-2hfz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:25:59.645976 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:59.645873    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c11a05bb-37ae-45ba-b257-7553097de947-maas-api-xks-ca" pod "c11a05bb-37ae-45ba-b257-7553097de947" (UID: "c11a05bb-37ae-45ba-b257-7553097de947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:25:59.745220 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:59.745158    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11a05bb-37ae-45ba-b257-7553097de947-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:25:59.745220 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:59.745182    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11a05bb-37ae-45ba-b257-7553097de947-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:25:59.745220 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:25:59.745192    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hfz8\" (UniqueName: \"kubernetes.io/projected/c11a05bb-37ae-45ba-b257-7553097de947-kube-api-access-2hfz8\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:26:00.442935 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:00.442851    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"
Nov 03 16:26:00.442935 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:00.442861    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb" event={"ID":"c11a05bb-37ae-45ba-b257-7553097de947","Type":"ContainerDied","Data":"170ffa70ad099fd747bb03d4bedcb344c650dc15073e06678f437c08ab2ff557"}
Nov 03 16:26:00.442935 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:00.442918    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170ffa70ad099fd747bb03d4bedcb344c650dc15073e06678f437c08ab2ff557"
Nov 03 16:26:03.035606 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:03.035574    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"]
Nov 03 16:26:03.040006 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:03.039978    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f65682-hdk9t"]
Nov 03 16:26:04.102138 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:04.102105    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aefed321-8510-4ca9-a066-1cbe448bf763" path="/var/lib/kubelet/pods/aefed321-8510-4ca9-a066-1cbe448bf763/volumes"
Nov 03 16:26:06.540501 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:06.540469    6506 scope.go:122] "RemoveContainer" containerID="e222c19ccc00799672b33bbd83abfcb17301e661b4c4f39f220edc3722681a1f"
Nov 03 16:26:09.986662 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:09.986625    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:26:16.027042 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.027002    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"]
Nov 03 16:26:16.027656 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.027633    6506 cpu_manager.go:414] "containerMap: removing container" podUID="c11a05bb-37ae-45ba-b257-7553097de947" containerName="revoke-keys"
Nov 03 16:26:16.027742 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.027660    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c11a05bb-37ae-45ba-b257-7553097de947" containerName="revoke-keys"
Nov 03 16:26:16.027815 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.027763    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="c11a05bb-37ae-45ba-b257-7553097de947" containerName="revoke-keys"
Nov 03 16:26:16.032382 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.032361    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"
Nov 03 16:26:16.040313 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.040289    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:26:16.040471 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.040378    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:26:16.043450 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.043428    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"]
Nov 03 16:26:16.082270 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.082238    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:26:16.083543 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.083521    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:26:16.115006 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.114984    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06bcbbde-9851-4747-8807-1089b1bb57c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-dzlvl\" (UID: \"06bcbbde-9851-4747-8807-1089b1bb57c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"
Nov 03 16:26:16.115149 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.115031    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06bcbbde-9851-4747-8807-1089b1bb57c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-dzlvl\" (UID: \"06bcbbde-9851-4747-8807-1089b1bb57c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"
Nov 03 16:26:16.115149 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.115106    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gvwv\" (UniqueName: \"kubernetes.io/projected/06bcbbde-9851-4747-8807-1089b1bb57c3-kube-api-access-7gvwv\") pod \"maas-api-revoke-sub-79805ca9-dzlvl\" (UID: \"06bcbbde-9851-4747-8807-1089b1bb57c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"
Nov 03 16:26:16.216215 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.216180    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06bcbbde-9851-4747-8807-1089b1bb57c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-dzlvl\" (UID: \"06bcbbde-9851-4747-8807-1089b1bb57c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"
Nov 03 16:26:16.216365 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.216241    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06bcbbde-9851-4747-8807-1089b1bb57c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-dzlvl\" (UID: \"06bcbbde-9851-4747-8807-1089b1bb57c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"
Nov 03 16:26:16.216365 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.216271    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gvwv\" (UniqueName: \"kubernetes.io/projected/06bcbbde-9851-4747-8807-1089b1bb57c3-kube-api-access-7gvwv\") pod \"maas-api-revoke-sub-79805ca9-dzlvl\" (UID: \"06bcbbde-9851-4747-8807-1089b1bb57c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"
Nov 03 16:26:16.216984 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.216963    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06bcbbde-9851-4747-8807-1089b1bb57c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-dzlvl\" (UID: \"06bcbbde-9851-4747-8807-1089b1bb57c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"
Nov 03 16:26:16.218724 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.218706    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06bcbbde-9851-4747-8807-1089b1bb57c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-dzlvl\" (UID: \"06bcbbde-9851-4747-8807-1089b1bb57c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"
Nov 03 16:26:16.231123 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.231096    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gvwv\" (UniqueName: \"kubernetes.io/projected/06bcbbde-9851-4747-8807-1089b1bb57c3-kube-api-access-7gvwv\") pod \"maas-api-revoke-sub-79805ca9-dzlvl\" (UID: \"06bcbbde-9851-4747-8807-1089b1bb57c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"
Nov 03 16:26:16.348207 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.348138    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"
Nov 03 16:26:16.626902 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:16.626872    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"]
Nov 03 16:26:16.628387 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:26:16.628352    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bcbbde_9851_4747_8807_1089b1bb57c3.slice/crio-fe4c058f1b8b150e078f47be29a86831b4cb99f1209a0cdb77c86292163936e3 WatchSource:0}: Error finding container fe4c058f1b8b150e078f47be29a86831b4cb99f1209a0cdb77c86292163936e3: Status 404 returned error can't find the container with id fe4c058f1b8b150e078f47be29a86831b4cb99f1209a0cdb77c86292163936e3
Nov 03 16:26:17.568788 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:17.568756    6506 generic.go:356] "Generic (PLEG): container finished" podID="06bcbbde-9851-4747-8807-1089b1bb57c3" containerID="2ee3b90beb1f7a1cdf98c13ea6ba7204a591b0cfcf1d287a662e61aa2a1be5c7" exitCode=0
Nov 03 16:26:17.569195 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:17.568834    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl" event={"ID":"06bcbbde-9851-4747-8807-1089b1bb57c3","Type":"ContainerDied","Data":"2ee3b90beb1f7a1cdf98c13ea6ba7204a591b0cfcf1d287a662e61aa2a1be5c7"}
Nov 03 16:26:17.569195 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:17.568896    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl" event={"ID":"06bcbbde-9851-4747-8807-1089b1bb57c3","Type":"ContainerStarted","Data":"fe4c058f1b8b150e078f47be29a86831b4cb99f1209a0cdb77c86292163936e3"}
Nov 03 16:26:18.703049 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:18.703023    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"
Nov 03 16:26:18.842676 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:18.842606    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06bcbbde-9851-4747-8807-1089b1bb57c3-kube-api-access-7gvwv\" (UniqueName: \"kubernetes.io/projected/06bcbbde-9851-4747-8807-1089b1bb57c3-kube-api-access-7gvwv\") pod \"06bcbbde-9851-4747-8807-1089b1bb57c3\" (UID: \"06bcbbde-9851-4747-8807-1089b1bb57c3\") "
Nov 03 16:26:18.842856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:18.842700    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06bcbbde-9851-4747-8807-1089b1bb57c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06bcbbde-9851-4747-8807-1089b1bb57c3-maas-api-xks-ca\") pod \"06bcbbde-9851-4747-8807-1089b1bb57c3\" (UID: \"06bcbbde-9851-4747-8807-1089b1bb57c3\") "
Nov 03 16:26:18.842856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:18.842736    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06bcbbde-9851-4747-8807-1089b1bb57c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06bcbbde-9851-4747-8807-1089b1bb57c3-maas-api-openshift-ca\") pod \"06bcbbde-9851-4747-8807-1089b1bb57c3\" (UID: \"06bcbbde-9851-4747-8807-1089b1bb57c3\") "
Nov 03 16:26:18.843125 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:18.843102    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06bcbbde-9851-4747-8807-1089b1bb57c3-maas-api-openshift-ca" pod "06bcbbde-9851-4747-8807-1089b1bb57c3" (UID: "06bcbbde-9851-4747-8807-1089b1bb57c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:26:18.844805 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:18.844763    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bcbbde-9851-4747-8807-1089b1bb57c3-kube-api-access-7gvwv" pod "06bcbbde-9851-4747-8807-1089b1bb57c3" (UID: "06bcbbde-9851-4747-8807-1089b1bb57c3"). InnerVolumeSpecName "kube-api-access-7gvwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:26:18.844893 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:18.844828    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06bcbbde-9851-4747-8807-1089b1bb57c3-maas-api-xks-ca" pod "06bcbbde-9851-4747-8807-1089b1bb57c3" (UID: "06bcbbde-9851-4747-8807-1089b1bb57c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:26:18.944159 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:18.944129    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gvwv\" (UniqueName: \"kubernetes.io/projected/06bcbbde-9851-4747-8807-1089b1bb57c3-kube-api-access-7gvwv\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:26:18.944159 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:18.944155    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06bcbbde-9851-4747-8807-1089b1bb57c3-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:26:18.944159 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:18.944164    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06bcbbde-9851-4747-8807-1089b1bb57c3-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:26:19.581322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:19.581289    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"
Nov 03 16:26:19.581508 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:19.581320    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl" event={"ID":"06bcbbde-9851-4747-8807-1089b1bb57c3","Type":"ContainerDied","Data":"fe4c058f1b8b150e078f47be29a86831b4cb99f1209a0cdb77c86292163936e3"}
Nov 03 16:26:19.581508 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:19.581375    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe4c058f1b8b150e078f47be29a86831b4cb99f1209a0cdb77c86292163936e3"
Nov 03 16:26:20.033768 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:20.033728    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"]
Nov 03 16:26:20.039747 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:20.039720    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea9526b-nqp7z"]
Nov 03 16:26:20.102722 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:20.102689    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac85523a-362a-4087-80c1-1db0c26d5f13" path="/var/lib/kubelet/pods/ac85523a-362a-4087-80c1-1db0c26d5f13/volumes"
Nov 03 16:26:28.795049 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:26:28.795009    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:26:31.138052 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:31.138011    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:26:31.138704 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:31.138651    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" podUID="13e172cb-f664-4da0-a922-08b5a0eb29d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:26:32.061712 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:32.061675    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"]
Nov 03 16:26:32.063940 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:32.063916    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-787a6067-wnw2r"]
Nov 03 16:26:32.103725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:32.103693    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="930af4d2-a222-47b3-b6d8-3aff6206f248" path="/var/lib/kubelet/pods/930af4d2-a222-47b3-b6d8-3aff6206f248/volumes"
Nov 03 16:26:33.955865 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:33.955827    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"]
Nov 03 16:26:33.956249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:33.956212    6506 cpu_manager.go:414] "containerMap: removing container" podUID="06bcbbde-9851-4747-8807-1089b1bb57c3" containerName="revoke-keys"
Nov 03 16:26:33.956249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:33.956223    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06bcbbde-9851-4747-8807-1089b1bb57c3" containerName="revoke-keys"
Nov 03 16:26:33.956321 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:33.956316    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="06bcbbde-9851-4747-8807-1089b1bb57c3" containerName="revoke-keys"
Nov 03 16:26:33.959659 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:33.959638    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"
Nov 03 16:26:33.967671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:33.967645    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:26:33.967822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:33.967727    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:26:33.967902 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:33.967851    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:26:33.967963 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:33.967923    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:26:33.970113 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:33.970087    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"]
Nov 03 16:26:34.014008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:34.013974    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cbba90-1807-412b-b826-1537db115c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gvwq6\" (UID: \"c1cbba90-1807-412b-b826-1537db115c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"
Nov 03 16:26:34.014168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:34.014060    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzctx\" (UniqueName: \"kubernetes.io/projected/c1cbba90-1807-412b-b826-1537db115c28-kube-api-access-kzctx\") pod \"maas-api-revoke-sub-0b7e8edb-gvwq6\" (UID: \"c1cbba90-1807-412b-b826-1537db115c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"
Nov 03 16:26:34.014168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:34.014139    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cbba90-1807-412b-b826-1537db115c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gvwq6\" (UID: \"c1cbba90-1807-412b-b826-1537db115c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"
Nov 03 16:26:34.115152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:34.115124    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cbba90-1807-412b-b826-1537db115c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gvwq6\" (UID: \"c1cbba90-1807-412b-b826-1537db115c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"
Nov 03 16:26:34.115323 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:34.115159    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzctx\" (UniqueName: \"kubernetes.io/projected/c1cbba90-1807-412b-b826-1537db115c28-kube-api-access-kzctx\") pod \"maas-api-revoke-sub-0b7e8edb-gvwq6\" (UID: \"c1cbba90-1807-412b-b826-1537db115c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"
Nov 03 16:26:34.115323 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:34.115185    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cbba90-1807-412b-b826-1537db115c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gvwq6\" (UID: \"c1cbba90-1807-412b-b826-1537db115c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"
Nov 03 16:26:34.115734 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:34.115703    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cbba90-1807-412b-b826-1537db115c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gvwq6\" (UID: \"c1cbba90-1807-412b-b826-1537db115c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"
Nov 03 16:26:34.117689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:34.117665    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cbba90-1807-412b-b826-1537db115c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gvwq6\" (UID: \"c1cbba90-1807-412b-b826-1537db115c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"
Nov 03 16:26:34.128030 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:34.128010    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzctx\" (UniqueName: \"kubernetes.io/projected/c1cbba90-1807-412b-b826-1537db115c28-kube-api-access-kzctx\") pod \"maas-api-revoke-sub-0b7e8edb-gvwq6\" (UID: \"c1cbba90-1807-412b-b826-1537db115c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"
Nov 03 16:26:34.273295 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:34.273273    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"
Nov 03 16:26:34.556846 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:34.556814    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"]
Nov 03 16:26:34.557417 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:26:34.557389    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cbba90_1807_412b_b826_1537db115c28.slice/crio-140fbb5c354d6f199065bf7bf8be5f095e44660753e91cf440b94d8506d74dda WatchSource:0}: Error finding container 140fbb5c354d6f199065bf7bf8be5f095e44660753e91cf440b94d8506d74dda: Status 404 returned error can't find the container with id 140fbb5c354d6f199065bf7bf8be5f095e44660753e91cf440b94d8506d74dda
Nov 03 16:26:34.712323 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:34.712295    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6" event={"ID":"c1cbba90-1807-412b-b826-1537db115c28","Type":"ContainerStarted","Data":"140fbb5c354d6f199065bf7bf8be5f095e44660753e91cf440b94d8506d74dda"}
Nov 03 16:26:35.719258 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:35.719173    6506 generic.go:356] "Generic (PLEG): container finished" podID="c1cbba90-1807-412b-b826-1537db115c28" containerID="9b6de4122ddf49897ead903cb8a58ef8154ee2102825fc1a4b36c2fcd4e65a36" exitCode=0
Nov 03 16:26:35.719258 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:35.719249    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6" event={"ID":"c1cbba90-1807-412b-b826-1537db115c28","Type":"ContainerDied","Data":"9b6de4122ddf49897ead903cb8a58ef8154ee2102825fc1a4b36c2fcd4e65a36"}
Nov 03 16:26:36.856008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:36.855978    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"
Nov 03 16:26:36.940359 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:36.940331    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1cbba90-1807-412b-b826-1537db115c28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cbba90-1807-412b-b826-1537db115c28-maas-api-xks-ca\") pod \"c1cbba90-1807-412b-b826-1537db115c28\" (UID: \"c1cbba90-1807-412b-b826-1537db115c28\") "
Nov 03 16:26:36.940509 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:36.940459    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1cbba90-1807-412b-b826-1537db115c28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cbba90-1807-412b-b826-1537db115c28-maas-api-openshift-ca\") pod \"c1cbba90-1807-412b-b826-1537db115c28\" (UID: \"c1cbba90-1807-412b-b826-1537db115c28\") "
Nov 03 16:26:36.940551 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:36.940519    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1cbba90-1807-412b-b826-1537db115c28-kube-api-access-kzctx\" (UniqueName: \"kubernetes.io/projected/c1cbba90-1807-412b-b826-1537db115c28-kube-api-access-kzctx\") pod \"c1cbba90-1807-412b-b826-1537db115c28\" (UID: \"c1cbba90-1807-412b-b826-1537db115c28\") "
Nov 03 16:26:36.940875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:36.940850    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1cbba90-1807-412b-b826-1537db115c28-maas-api-openshift-ca" pod "c1cbba90-1807-412b-b826-1537db115c28" (UID: "c1cbba90-1807-412b-b826-1537db115c28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:26:36.942642 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:36.942615    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1cbba90-1807-412b-b826-1537db115c28-maas-api-xks-ca" pod "c1cbba90-1807-412b-b826-1537db115c28" (UID: "c1cbba90-1807-412b-b826-1537db115c28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:26:36.942766 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:36.942745    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cbba90-1807-412b-b826-1537db115c28-kube-api-access-kzctx" pod "c1cbba90-1807-412b-b826-1537db115c28" (UID: "c1cbba90-1807-412b-b826-1537db115c28"). InnerVolumeSpecName "kube-api-access-kzctx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:26:37.042127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:37.042099    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cbba90-1807-412b-b826-1537db115c28-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:26:37.042127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:37.042125    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzctx\" (UniqueName: \"kubernetes.io/projected/c1cbba90-1807-412b-b826-1537db115c28-kube-api-access-kzctx\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:26:37.042296 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:37.042136    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cbba90-1807-412b-b826-1537db115c28-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:26:37.731975 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:37.731939    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6" event={"ID":"c1cbba90-1807-412b-b826-1537db115c28","Type":"ContainerDied","Data":"140fbb5c354d6f199065bf7bf8be5f095e44660753e91cf440b94d8506d74dda"}
Nov 03 16:26:37.731975 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:37.731968    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"
Nov 03 16:26:37.732183 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:37.731999    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="140fbb5c354d6f199065bf7bf8be5f095e44660753e91cf440b94d8506d74dda"
Nov 03 16:26:40.033891 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:40.033862    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"]
Nov 03 16:26:40.046252 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:40.046222    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf8d3d4-mdl7j"]
Nov 03 16:26:40.102206 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:40.102174    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="854b1626-e3b6-43dc-82a2-d83512df2718" path="/var/lib/kubelet/pods/854b1626-e3b6-43dc-82a2-d83512df2718/volumes"
Nov 03 16:26:44.721270 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.721235    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"]
Nov 03 16:26:44.721687 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.721621    6506 cpu_manager.go:414] "containerMap: removing container" podUID="c1cbba90-1807-412b-b826-1537db115c28" containerName="revoke-keys"
Nov 03 16:26:44.721687 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.721635    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1cbba90-1807-412b-b826-1537db115c28" containerName="revoke-keys"
Nov 03 16:26:44.721768 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.721719    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1cbba90-1807-412b-b826-1537db115c28" containerName="revoke-keys"
Nov 03 16:26:44.725232 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.725214    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"
Nov 03 16:26:44.730996 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.730974    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:26:44.731174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.731161    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:26:44.731244 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.731207    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:26:44.731244 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.731211    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:26:44.744141 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.744119    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"]
Nov 03 16:26:44.815623 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.815595    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktn87\" (UniqueName: \"kubernetes.io/projected/48c2a112-7c75-4a3a-a732-dffd5ccecca6-kube-api-access-ktn87\") pod \"maas-api-revoke-sub-3d4156f1-bl6r6\" (UID: \"48c2a112-7c75-4a3a-a732-dffd5ccecca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"
Nov 03 16:26:44.815762 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.815641    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c2a112-7c75-4a3a-a732-dffd5ccecca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bl6r6\" (UID: \"48c2a112-7c75-4a3a-a732-dffd5ccecca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"
Nov 03 16:26:44.815842 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.815757    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c2a112-7c75-4a3a-a732-dffd5ccecca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bl6r6\" (UID: \"48c2a112-7c75-4a3a-a732-dffd5ccecca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"
Nov 03 16:26:44.916880 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.916849    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktn87\" (UniqueName: \"kubernetes.io/projected/48c2a112-7c75-4a3a-a732-dffd5ccecca6-kube-api-access-ktn87\") pod \"maas-api-revoke-sub-3d4156f1-bl6r6\" (UID: \"48c2a112-7c75-4a3a-a732-dffd5ccecca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"
Nov 03 16:26:44.917025 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.916908    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c2a112-7c75-4a3a-a732-dffd5ccecca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bl6r6\" (UID: \"48c2a112-7c75-4a3a-a732-dffd5ccecca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"
Nov 03 16:26:44.917025 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.916977    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c2a112-7c75-4a3a-a732-dffd5ccecca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bl6r6\" (UID: \"48c2a112-7c75-4a3a-a732-dffd5ccecca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"
Nov 03 16:26:44.917603 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.917573    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c2a112-7c75-4a3a-a732-dffd5ccecca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bl6r6\" (UID: \"48c2a112-7c75-4a3a-a732-dffd5ccecca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"
Nov 03 16:26:44.919271 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.919253    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c2a112-7c75-4a3a-a732-dffd5ccecca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bl6r6\" (UID: \"48c2a112-7c75-4a3a-a732-dffd5ccecca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"
Nov 03 16:26:44.931809 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:44.931767    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktn87\" (UniqueName: \"kubernetes.io/projected/48c2a112-7c75-4a3a-a732-dffd5ccecca6-kube-api-access-ktn87\") pod \"maas-api-revoke-sub-3d4156f1-bl6r6\" (UID: \"48c2a112-7c75-4a3a-a732-dffd5ccecca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"
Nov 03 16:26:45.038299 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:45.038270    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"
Nov 03 16:26:45.316033 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:45.316006    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"]
Nov 03 16:26:45.317281 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:26:45.317250    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c2a112_7c75_4a3a_a732_dffd5ccecca6.slice/crio-911778b60946e432b5870e25d1c5f88364e20a2cfa7ba61f0ce7479361a0f2db WatchSource:0}: Error finding container 911778b60946e432b5870e25d1c5f88364e20a2cfa7ba61f0ce7479361a0f2db: Status 404 returned error can't find the container with id 911778b60946e432b5870e25d1c5f88364e20a2cfa7ba61f0ce7479361a0f2db
Nov 03 16:26:45.780416 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:45.780380    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6" event={"ID":"48c2a112-7c75-4a3a-a732-dffd5ccecca6","Type":"ContainerStarted","Data":"154c5d4c9abc26a10779548b367f1a0f5a303e310f2913ec58a59e15bd98ac9a"}
Nov 03 16:26:45.780834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:45.780444    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6" event={"ID":"48c2a112-7c75-4a3a-a732-dffd5ccecca6","Type":"ContainerStarted","Data":"911778b60946e432b5870e25d1c5f88364e20a2cfa7ba61f0ce7479361a0f2db"}
Nov 03 16:26:46.787117 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:46.787079    6506 generic.go:356] "Generic (PLEG): container finished" podID="48c2a112-7c75-4a3a-a732-dffd5ccecca6" containerID="154c5d4c9abc26a10779548b367f1a0f5a303e310f2913ec58a59e15bd98ac9a" exitCode=0
Nov 03 16:26:46.787512 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:46.787172    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6" event={"ID":"48c2a112-7c75-4a3a-a732-dffd5ccecca6","Type":"ContainerDied","Data":"154c5d4c9abc26a10779548b367f1a0f5a303e310f2913ec58a59e15bd98ac9a"}
Nov 03 16:26:47.932722 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:47.932699    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"
Nov 03 16:26:48.048514 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:48.048483    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c2a112-7c75-4a3a-a732-dffd5ccecca6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c2a112-7c75-4a3a-a732-dffd5ccecca6-maas-api-openshift-ca\") pod \"48c2a112-7c75-4a3a-a732-dffd5ccecca6\" (UID: \"48c2a112-7c75-4a3a-a732-dffd5ccecca6\") "
Nov 03 16:26:48.048701 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:48.048575    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48c2a112-7c75-4a3a-a732-dffd5ccecca6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c2a112-7c75-4a3a-a732-dffd5ccecca6-maas-api-xks-ca\") pod \"48c2a112-7c75-4a3a-a732-dffd5ccecca6\" (UID: \"48c2a112-7c75-4a3a-a732-dffd5ccecca6\") "
Nov 03 16:26:48.048701 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:48.048608    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c2a112-7c75-4a3a-a732-dffd5ccecca6-kube-api-access-ktn87\" (UniqueName: \"kubernetes.io/projected/48c2a112-7c75-4a3a-a732-dffd5ccecca6-kube-api-access-ktn87\") pod \"48c2a112-7c75-4a3a-a732-dffd5ccecca6\" (UID: \"48c2a112-7c75-4a3a-a732-dffd5ccecca6\") "
Nov 03 16:26:48.048930 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:48.048908    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c2a112-7c75-4a3a-a732-dffd5ccecca6-maas-api-openshift-ca" pod "48c2a112-7c75-4a3a-a732-dffd5ccecca6" (UID: "48c2a112-7c75-4a3a-a732-dffd5ccecca6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:26:48.050817 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:48.050743    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c2a112-7c75-4a3a-a732-dffd5ccecca6-maas-api-xks-ca" pod "48c2a112-7c75-4a3a-a732-dffd5ccecca6" (UID: "48c2a112-7c75-4a3a-a732-dffd5ccecca6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:26:48.050817 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:48.050781    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c2a112-7c75-4a3a-a732-dffd5ccecca6-kube-api-access-ktn87" pod "48c2a112-7c75-4a3a-a732-dffd5ccecca6" (UID: "48c2a112-7c75-4a3a-a732-dffd5ccecca6"). InnerVolumeSpecName "kube-api-access-ktn87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:26:48.149403 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:48.149371    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c2a112-7c75-4a3a-a732-dffd5ccecca6-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:26:48.149403 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:48.149404    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c2a112-7c75-4a3a-a732-dffd5ccecca6-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:26:48.149566 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:48.149419    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktn87\" (UniqueName: \"kubernetes.io/projected/48c2a112-7c75-4a3a-a732-dffd5ccecca6-kube-api-access-ktn87\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:26:48.807495 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:48.807465    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"
Nov 03 16:26:48.807495 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:48.807488    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6" event={"ID":"48c2a112-7c75-4a3a-a732-dffd5ccecca6","Type":"ContainerDied","Data":"911778b60946e432b5870e25d1c5f88364e20a2cfa7ba61f0ce7479361a0f2db"}
Nov 03 16:26:48.807761 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:48.807533    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="911778b60946e432b5870e25d1c5f88364e20a2cfa7ba61f0ce7479361a0f2db"
Nov 03 16:26:56.026179 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:56.026143    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"]
Nov 03 16:26:56.029555 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:56.029533    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108034f4-gl9h8"]
Nov 03 16:26:56.103119 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:26:56.103073    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc0055d-6da5-4866-915c-b81476bfecbd" path="/var/lib/kubelet/pods/bcc0055d-6da5-4866-915c-b81476bfecbd/volumes"
Nov 03 16:26:58.803191 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:26:58.803147    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:27:00.034819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:00.033783    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"]
Nov 03 16:27:00.066819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:00.065108    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4576c79-mfpgz"]
Nov 03 16:27:00.102159 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:00.102131    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a4e8de-d68f-4726-8699-d452f998cc64" path="/var/lib/kubelet/pods/52a4e8de-d68f-4726-8699-d452f998cc64/volumes"
Nov 03 16:27:01.138040 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:01.137986    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:27:01.138560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:01.138096    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" podUID="13e172cb-f664-4da0-a922-08b5a0eb29d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:27:02.031783 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:02.031743    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"]
Nov 03 16:27:02.034182 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:02.034151    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d7abf06-4jdz2"]
Nov 03 16:27:02.102984 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:02.102950    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc" path="/var/lib/kubelet/pods/b2f7f17c-0ba8-4117-8b7c-3a4627f36bcc/volumes"
Nov 03 16:27:06.094005 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.093971    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"]
Nov 03 16:27:06.103494 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:27:06.103449    6506 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-3d4156f1-bl6r6\" is forbidden: User \"system:node:ip-10-0-137-159.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-137-159.us-east-2.compute.internal' and this object" podUID="48c2a112-7c75-4a3a-a732-dffd5ccecca6" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"
Nov 03 16:27:06.107348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.107322    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bl6r6"]
Nov 03 16:27:06.126270 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.126246    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"]
Nov 03 16:27:06.126616 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.126603    6506 cpu_manager.go:414] "containerMap: removing container" podUID="48c2a112-7c75-4a3a-a732-dffd5ccecca6" containerName="revoke-keys"
Nov 03 16:27:06.126666 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.126618    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c2a112-7c75-4a3a-a732-dffd5ccecca6" containerName="revoke-keys"
Nov 03 16:27:06.126712 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.126701    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c2a112-7c75-4a3a-a732-dffd5ccecca6" containerName="revoke-keys"
Nov 03 16:27:06.130999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.130984    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"
Nov 03 16:27:06.140017 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.136928    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:27:06.140017 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.138613    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:27:06.140017 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.138705    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:27:06.140017 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.138754    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:27:06.140817 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.140766    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"]
Nov 03 16:27:06.248844 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.248787    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd555c7-745a-4cf4-9bae-13e2793ff62b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ptr25\" (UID: \"3bd555c7-745a-4cf4-9bae-13e2793ff62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"
Nov 03 16:27:06.249014 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.248923    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6krw\" (UniqueName: \"kubernetes.io/projected/3bd555c7-745a-4cf4-9bae-13e2793ff62b-kube-api-access-b6krw\") pod \"maas-api-revoke-sub-3d4156f1-ptr25\" (UID: \"3bd555c7-745a-4cf4-9bae-13e2793ff62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"
Nov 03 16:27:06.249014 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.248979    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd555c7-745a-4cf4-9bae-13e2793ff62b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ptr25\" (UID: \"3bd555c7-745a-4cf4-9bae-13e2793ff62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"
Nov 03 16:27:06.350210 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.350127    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd555c7-745a-4cf4-9bae-13e2793ff62b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ptr25\" (UID: \"3bd555c7-745a-4cf4-9bae-13e2793ff62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"
Nov 03 16:27:06.350366 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.350212    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6krw\" (UniqueName: \"kubernetes.io/projected/3bd555c7-745a-4cf4-9bae-13e2793ff62b-kube-api-access-b6krw\") pod \"maas-api-revoke-sub-3d4156f1-ptr25\" (UID: \"3bd555c7-745a-4cf4-9bae-13e2793ff62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"
Nov 03 16:27:06.350366 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.350268    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd555c7-745a-4cf4-9bae-13e2793ff62b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ptr25\" (UID: \"3bd555c7-745a-4cf4-9bae-13e2793ff62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"
Nov 03 16:27:06.350929 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.350871    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd555c7-745a-4cf4-9bae-13e2793ff62b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ptr25\" (UID: \"3bd555c7-745a-4cf4-9bae-13e2793ff62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"
Nov 03 16:27:06.352374 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.352353    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd555c7-745a-4cf4-9bae-13e2793ff62b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ptr25\" (UID: \"3bd555c7-745a-4cf4-9bae-13e2793ff62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"
Nov 03 16:27:06.369993 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.369970    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6krw\" (UniqueName: \"kubernetes.io/projected/3bd555c7-745a-4cf4-9bae-13e2793ff62b-kube-api-access-b6krw\") pod \"maas-api-revoke-sub-3d4156f1-ptr25\" (UID: \"3bd555c7-745a-4cf4-9bae-13e2793ff62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"
Nov 03 16:27:06.445558 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.445530    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"
Nov 03 16:27:06.567172 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.567140    6506 scope.go:122] "RemoveContainer" containerID="8c49a6b6d0b5142f6fc1d722687037b5db8c3b2558f10a1f5f336925d3c11cb5"
Nov 03 16:27:06.576287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.576268    6506 scope.go:122] "RemoveContainer" containerID="e6b388ec7a670c66516bcfd004850409af52e2bf582307e116b46c4b9926727c"
Nov 03 16:27:06.585084 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.585064    6506 scope.go:122] "RemoveContainer" containerID="c00bb1ceaa16a30289f62347599581dfb395e5d43f1a9b980a2e3bc22a0ddd90"
Nov 03 16:27:06.723899 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.723876    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"]
Nov 03 16:27:06.725304 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:27:06.725267    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd555c7_745a_4cf4_9bae_13e2793ff62b.slice/crio-78e52524b241c2078e10ac211ce37c91719727b57dd79bed1b68529e43c0a25b WatchSource:0}: Error finding container 78e52524b241c2078e10ac211ce37c91719727b57dd79bed1b68529e43c0a25b: Status 404 returned error can't find the container with id 78e52524b241c2078e10ac211ce37c91719727b57dd79bed1b68529e43c0a25b
Nov 03 16:27:06.940220 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:06.940184    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25" event={"ID":"3bd555c7-745a-4cf4-9bae-13e2793ff62b","Type":"ContainerStarted","Data":"78e52524b241c2078e10ac211ce37c91719727b57dd79bed1b68529e43c0a25b"}
Nov 03 16:27:07.947440 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:07.947401    6506 generic.go:356] "Generic (PLEG): container finished" podID="3bd555c7-745a-4cf4-9bae-13e2793ff62b" containerID="41415ea4bfb89dd1ab53e8207143875a235d621905bc1adc01192c50d3be1813" exitCode=0
Nov 03 16:27:07.947891 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:07.947487    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25" event={"ID":"3bd555c7-745a-4cf4-9bae-13e2793ff62b","Type":"ContainerDied","Data":"41415ea4bfb89dd1ab53e8207143875a235d621905bc1adc01192c50d3be1813"}
Nov 03 16:27:08.104092 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:08.104064    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c2a112-7c75-4a3a-a732-dffd5ccecca6" path="/var/lib/kubelet/pods/48c2a112-7c75-4a3a-a732-dffd5ccecca6/volumes"
Nov 03 16:27:09.032058 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:09.032029    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"]
Nov 03 16:27:09.033704 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:09.033677    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09409a22-qpsp9"]
Nov 03 16:27:09.085909 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:09.085887    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"
Nov 03 16:27:09.177386 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:09.177361    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bd555c7-745a-4cf4-9bae-13e2793ff62b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd555c7-745a-4cf4-9bae-13e2793ff62b-maas-api-xks-ca\") pod \"3bd555c7-745a-4cf4-9bae-13e2793ff62b\" (UID: \"3bd555c7-745a-4cf4-9bae-13e2793ff62b\") "
Nov 03 16:27:09.177543 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:09.177449    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bd555c7-745a-4cf4-9bae-13e2793ff62b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd555c7-745a-4cf4-9bae-13e2793ff62b-maas-api-openshift-ca\") pod \"3bd555c7-745a-4cf4-9bae-13e2793ff62b\" (UID: \"3bd555c7-745a-4cf4-9bae-13e2793ff62b\") "
Nov 03 16:27:09.177543 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:09.177487    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd555c7-745a-4cf4-9bae-13e2793ff62b-kube-api-access-b6krw\" (UniqueName: \"kubernetes.io/projected/3bd555c7-745a-4cf4-9bae-13e2793ff62b-kube-api-access-b6krw\") pod \"3bd555c7-745a-4cf4-9bae-13e2793ff62b\" (UID: \"3bd555c7-745a-4cf4-9bae-13e2793ff62b\") "
Nov 03 16:27:09.177847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:09.177822    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd555c7-745a-4cf4-9bae-13e2793ff62b-maas-api-openshift-ca" pod "3bd555c7-745a-4cf4-9bae-13e2793ff62b" (UID: "3bd555c7-745a-4cf4-9bae-13e2793ff62b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:27:09.179612 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:09.179580    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd555c7-745a-4cf4-9bae-13e2793ff62b-maas-api-xks-ca" pod "3bd555c7-745a-4cf4-9bae-13e2793ff62b" (UID: "3bd555c7-745a-4cf4-9bae-13e2793ff62b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:27:09.179723 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:09.179663    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd555c7-745a-4cf4-9bae-13e2793ff62b-kube-api-access-b6krw" pod "3bd555c7-745a-4cf4-9bae-13e2793ff62b" (UID: "3bd555c7-745a-4cf4-9bae-13e2793ff62b"). InnerVolumeSpecName "kube-api-access-b6krw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:27:09.279008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:09.278975    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6krw\" (UniqueName: \"kubernetes.io/projected/3bd555c7-745a-4cf4-9bae-13e2793ff62b-kube-api-access-b6krw\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:09.279008 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:09.279007    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd555c7-745a-4cf4-9bae-13e2793ff62b-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:09.279165 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:09.279021    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd555c7-745a-4cf4-9bae-13e2793ff62b-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:09.959442 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:09.959404    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25" event={"ID":"3bd555c7-745a-4cf4-9bae-13e2793ff62b","Type":"ContainerDied","Data":"78e52524b241c2078e10ac211ce37c91719727b57dd79bed1b68529e43c0a25b"}
Nov 03 16:27:09.959442 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:09.959424    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"
Nov 03 16:27:09.959704 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:09.959454    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78e52524b241c2078e10ac211ce37c91719727b57dd79bed1b68529e43c0a25b"
Nov 03 16:27:10.101645 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:10.101613    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874bd46d-b2ff-4799-8d5c-6e55b8f79066" path="/var/lib/kubelet/pods/874bd46d-b2ff-4799-8d5c-6e55b8f79066/volumes"
Nov 03 16:27:11.926069 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:11.926038    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:27:12.084647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:12.084614    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:27:19.030485 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:19.030397    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"]
Nov 03 16:27:19.033823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:19.033780    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f01d0ba-rpgtj"]
Nov 03 16:27:20.102398 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:20.102364    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288ab83a-1d57-4f44-800f-a9850aed7772" path="/var/lib/kubelet/pods/288ab83a-1d57-4f44-800f-a9850aed7772/volumes"
Nov 03 16:27:28.873732 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:27:28.873689    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:27:32.031125 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:32.031086    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"]
Nov 03 16:27:32.033363 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:32.033337    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d89707d-lh546"]
Nov 03 16:27:32.102111 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:32.102076    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f093183a-3336-4dce-8945-f907211722f4" path="/var/lib/kubelet/pods/f093183a-3336-4dce-8945-f907211722f4/volumes"
Nov 03 16:27:35.030266 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.030214    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"]
Nov 03 16:27:35.030830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.030786    6506 cpu_manager.go:414] "containerMap: removing container" podUID="3bd555c7-745a-4cf4-9bae-13e2793ff62b" containerName="revoke-keys"
Nov 03 16:27:35.030929 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.030828    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd555c7-745a-4cf4-9bae-13e2793ff62b" containerName="revoke-keys"
Nov 03 16:27:35.030989 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.030946    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd555c7-745a-4cf4-9bae-13e2793ff62b" containerName="revoke-keys"
Nov 03 16:27:35.034247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.034226    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"
Nov 03 16:27:35.040311 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.040292    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:27:35.040411 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.040331    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:27:35.040463 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.040417    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:27:35.041568 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.041552    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:27:35.047032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.047009    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"]
Nov 03 16:27:35.142545 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.142516    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zcvg\" (UniqueName: \"kubernetes.io/projected/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-kube-api-access-2zcvg\") pod \"maas-api-revoke-sub-e2fc9c5b-xv69m\" (UID: \"5fdb6dd9-c8da-430e-8571-a8bd31214c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"
Nov 03 16:27:35.142723 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.142692    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-xv69m\" (UID: \"5fdb6dd9-c8da-430e-8571-a8bd31214c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"
Nov 03 16:27:35.142785 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.142769    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-xv69m\" (UID: \"5fdb6dd9-c8da-430e-8571-a8bd31214c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"
Nov 03 16:27:35.244195 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.244162    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-xv69m\" (UID: \"5fdb6dd9-c8da-430e-8571-a8bd31214c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"
Nov 03 16:27:35.244405 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.244223    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-xv69m\" (UID: \"5fdb6dd9-c8da-430e-8571-a8bd31214c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"
Nov 03 16:27:35.244405 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.244248    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zcvg\" (UniqueName: \"kubernetes.io/projected/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-kube-api-access-2zcvg\") pod \"maas-api-revoke-sub-e2fc9c5b-xv69m\" (UID: \"5fdb6dd9-c8da-430e-8571-a8bd31214c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"
Nov 03 16:27:35.244949 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.244931    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-xv69m\" (UID: \"5fdb6dd9-c8da-430e-8571-a8bd31214c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"
Nov 03 16:27:35.246652 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.246633    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-xv69m\" (UID: \"5fdb6dd9-c8da-430e-8571-a8bd31214c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"
Nov 03 16:27:35.257440 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.257420    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zcvg\" (UniqueName: \"kubernetes.io/projected/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-kube-api-access-2zcvg\") pod \"maas-api-revoke-sub-e2fc9c5b-xv69m\" (UID: \"5fdb6dd9-c8da-430e-8571-a8bd31214c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"
Nov 03 16:27:35.348140 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.348079    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"
Nov 03 16:27:35.630947 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:35.630918    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"]
Nov 03 16:27:35.632485 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:27:35.632458    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdb6dd9_c8da_430e_8571_a8bd31214c2f.slice/crio-8bd5cc00c8817fb308966983167ae8105c01665ee828c3b6ef8db35001f5455e WatchSource:0}: Error finding container 8bd5cc00c8817fb308966983167ae8105c01665ee828c3b6ef8db35001f5455e: Status 404 returned error can't find the container with id 8bd5cc00c8817fb308966983167ae8105c01665ee828c3b6ef8db35001f5455e
Nov 03 16:27:36.124632 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:36.124593    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m" event={"ID":"5fdb6dd9-c8da-430e-8571-a8bd31214c2f","Type":"ContainerStarted","Data":"38641aff7c507c09270d61fe047cd54a800e8d570f338d7f8ab7fb2a2ecab382"}
Nov 03 16:27:36.125040 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:36.124656    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m" event={"ID":"5fdb6dd9-c8da-430e-8571-a8bd31214c2f","Type":"ContainerStarted","Data":"8bd5cc00c8817fb308966983167ae8105c01665ee828c3b6ef8db35001f5455e"}
Nov 03 16:27:36.145294 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:36.145247    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m" podStartSLOduration=1.145232646 podStartE2EDuration="1.145232646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:27:36.143724801 +0000 UTC m=+1470.157122950" watchObservedRunningTime="2026-11-03 16:27:36.145232646 +0000 UTC m=+1470.158630791"
Nov 03 16:27:37.131752 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:37.131710    6506 generic.go:356] "Generic (PLEG): container finished" podID="5fdb6dd9-c8da-430e-8571-a8bd31214c2f" containerID="38641aff7c507c09270d61fe047cd54a800e8d570f338d7f8ab7fb2a2ecab382" exitCode=0
Nov 03 16:27:37.132156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:37.131766    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m" event={"ID":"5fdb6dd9-c8da-430e-8571-a8bd31214c2f","Type":"ContainerDied","Data":"38641aff7c507c09270d61fe047cd54a800e8d570f338d7f8ab7fb2a2ecab382"}
Nov 03 16:27:38.269242 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:38.269209    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"
Nov 03 16:27:38.371175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:38.371144    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-maas-api-xks-ca\") pod \"5fdb6dd9-c8da-430e-8571-a8bd31214c2f\" (UID: \"5fdb6dd9-c8da-430e-8571-a8bd31214c2f\") "
Nov 03 16:27:38.371340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:38.371187    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-maas-api-openshift-ca\") pod \"5fdb6dd9-c8da-430e-8571-a8bd31214c2f\" (UID: \"5fdb6dd9-c8da-430e-8571-a8bd31214c2f\") "
Nov 03 16:27:38.371340 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:38.371273    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-kube-api-access-2zcvg\" (UniqueName: \"kubernetes.io/projected/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-kube-api-access-2zcvg\") pod \"5fdb6dd9-c8da-430e-8571-a8bd31214c2f\" (UID: \"5fdb6dd9-c8da-430e-8571-a8bd31214c2f\") "
Nov 03 16:27:38.371569 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:38.371548    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-maas-api-openshift-ca" pod "5fdb6dd9-c8da-430e-8571-a8bd31214c2f" (UID: "5fdb6dd9-c8da-430e-8571-a8bd31214c2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:27:38.373397 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:38.373375    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-kube-api-access-2zcvg" pod "5fdb6dd9-c8da-430e-8571-a8bd31214c2f" (UID: "5fdb6dd9-c8da-430e-8571-a8bd31214c2f"). InnerVolumeSpecName "kube-api-access-2zcvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:27:38.373510 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:38.373491    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-maas-api-xks-ca" pod "5fdb6dd9-c8da-430e-8571-a8bd31214c2f" (UID: "5fdb6dd9-c8da-430e-8571-a8bd31214c2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:27:38.472488 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:38.472420    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:38.472488 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:38.472448    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:38.472488 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:38.472459    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zcvg\" (UniqueName: \"kubernetes.io/projected/5fdb6dd9-c8da-430e-8571-a8bd31214c2f-kube-api-access-2zcvg\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:39.145053 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:39.145015    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m" event={"ID":"5fdb6dd9-c8da-430e-8571-a8bd31214c2f","Type":"ContainerDied","Data":"8bd5cc00c8817fb308966983167ae8105c01665ee828c3b6ef8db35001f5455e"}
Nov 03 16:27:39.145053 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:39.145060    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bd5cc00c8817fb308966983167ae8105c01665ee828c3b6ef8db35001f5455e"
Nov 03 16:27:39.145277 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:39.145027    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"
Nov 03 16:27:42.073063 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:42.073013    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:27:45.255773 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.255737    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"]
Nov 03 16:27:45.256181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.256143    6506 cpu_manager.go:414] "containerMap: removing container" podUID="5fdb6dd9-c8da-430e-8571-a8bd31214c2f" containerName="revoke-keys"
Nov 03 16:27:45.256181 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.256159    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fdb6dd9-c8da-430e-8571-a8bd31214c2f" containerName="revoke-keys"
Nov 03 16:27:45.256271 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.256257    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fdb6dd9-c8da-430e-8571-a8bd31214c2f" containerName="revoke-keys"
Nov 03 16:27:45.259822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.259787    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"
Nov 03 16:27:45.266441 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.266421    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:27:45.266678 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.266656    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:27:45.266779 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.266755    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:27:45.266958 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.266781    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:27:45.274718 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.274696    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"]
Nov 03 16:27:45.331897 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.331870    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d917a62-fdad-458f-abcb-7267f36b02f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-7mf96\" (UID: \"4d917a62-fdad-458f-abcb-7267f36b02f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"
Nov 03 16:27:45.332067 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.331939    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d917a62-fdad-458f-abcb-7267f36b02f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-7mf96\" (UID: \"4d917a62-fdad-458f-abcb-7267f36b02f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"
Nov 03 16:27:45.332067 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.332024    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887x2\" (UniqueName: \"kubernetes.io/projected/4d917a62-fdad-458f-abcb-7267f36b02f7-kube-api-access-887x2\") pod \"maas-api-revoke-sub-5a29efb3-7mf96\" (UID: \"4d917a62-fdad-458f-abcb-7267f36b02f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"
Nov 03 16:27:45.432495 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.432463    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d917a62-fdad-458f-abcb-7267f36b02f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-7mf96\" (UID: \"4d917a62-fdad-458f-abcb-7267f36b02f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"
Nov 03 16:27:45.432679 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.432523    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887x2\" (UniqueName: \"kubernetes.io/projected/4d917a62-fdad-458f-abcb-7267f36b02f7-kube-api-access-887x2\") pod \"maas-api-revoke-sub-5a29efb3-7mf96\" (UID: \"4d917a62-fdad-458f-abcb-7267f36b02f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"
Nov 03 16:27:45.432679 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.432584    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d917a62-fdad-458f-abcb-7267f36b02f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-7mf96\" (UID: \"4d917a62-fdad-458f-abcb-7267f36b02f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"
Nov 03 16:27:45.433318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.433298    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d917a62-fdad-458f-abcb-7267f36b02f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-7mf96\" (UID: \"4d917a62-fdad-458f-abcb-7267f36b02f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"
Nov 03 16:27:45.435040 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.435018    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d917a62-fdad-458f-abcb-7267f36b02f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-7mf96\" (UID: \"4d917a62-fdad-458f-abcb-7267f36b02f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"
Nov 03 16:27:45.446203 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.446184    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887x2\" (UniqueName: \"kubernetes.io/projected/4d917a62-fdad-458f-abcb-7267f36b02f7-kube-api-access-887x2\") pod \"maas-api-revoke-sub-5a29efb3-7mf96\" (UID: \"4d917a62-fdad-458f-abcb-7267f36b02f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"
Nov 03 16:27:45.574488 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.574459    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"
Nov 03 16:27:45.851938 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:45.851916    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"]
Nov 03 16:27:45.854330 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:27:45.854301    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d917a62_fdad_458f_abcb_7267f36b02f7.slice/crio-1d451c84d274203d9980223840f9255cfb03867789896b077c7f85be680e9bde WatchSource:0}: Error finding container 1d451c84d274203d9980223840f9255cfb03867789896b077c7f85be680e9bde: Status 404 returned error can't find the container with id 1d451c84d274203d9980223840f9255cfb03867789896b077c7f85be680e9bde
Nov 03 16:27:46.187738 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:46.187703    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96" event={"ID":"4d917a62-fdad-458f-abcb-7267f36b02f7","Type":"ContainerStarted","Data":"35fd670030fe8c37aeb66bb9db7595b4df45581494c0c8eb9f198af832c989d8"}
Nov 03 16:27:46.187900 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:46.187760    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96" event={"ID":"4d917a62-fdad-458f-abcb-7267f36b02f7","Type":"ContainerStarted","Data":"1d451c84d274203d9980223840f9255cfb03867789896b077c7f85be680e9bde"}
Nov 03 16:27:46.230645 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:46.230591    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96" podStartSLOduration=1.230571788 podStartE2EDuration="1.230571788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:27:46.226780732 +0000 UTC m=+1480.240178881" watchObservedRunningTime="2026-11-03 16:27:46.230571788 +0000 UTC m=+1480.243969935"
Nov 03 16:27:47.055732 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.055691    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"]
Nov 03 16:27:47.056277 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.055942    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7" podUID="ab567ff1-957a-4316-9939-f0977aeecde7" containerName="discovery" containerID="cri-o://4c66698a2713b66534c4ef978011b746a08499f03bbcf22551930831cf29800a" gracePeriod=30
Nov 03 16:27:47.197298 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.197262    6506 generic.go:356] "Generic (PLEG): container finished" podID="4d917a62-fdad-458f-abcb-7267f36b02f7" containerID="35fd670030fe8c37aeb66bb9db7595b4df45581494c0c8eb9f198af832c989d8" exitCode=0
Nov 03 16:27:47.197484 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.197347    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96" event={"ID":"4d917a62-fdad-458f-abcb-7267f36b02f7","Type":"ContainerDied","Data":"35fd670030fe8c37aeb66bb9db7595b4df45581494c0c8eb9f198af832c989d8"}
Nov 03 16:27:47.199436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.199413    6506 generic.go:356] "Generic (PLEG): container finished" podID="ab567ff1-957a-4316-9939-f0977aeecde7" containerID="4c66698a2713b66534c4ef978011b746a08499f03bbcf22551930831cf29800a" exitCode=0
Nov 03 16:27:47.199581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.199504    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7" event={"ID":"ab567ff1-957a-4316-9939-f0977aeecde7","Type":"ContainerDied","Data":"4c66698a2713b66534c4ef978011b746a08499f03bbcf22551930831cf29800a"}
Nov 03 16:27:47.492823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.492174    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:27:47.588828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.587144    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab567ff1-957a-4316-9939-f0977aeecde7-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab567ff1-957a-4316-9939-f0977aeecde7-local-certs\") pod \"ab567ff1-957a-4316-9939-f0977aeecde7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") "
Nov 03 16:27:47.588828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.587204    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab567ff1-957a-4316-9939-f0977aeecde7-istio-token\" (UniqueName: \"kubernetes.io/projected/ab567ff1-957a-4316-9939-f0977aeecde7-istio-token\") pod \"ab567ff1-957a-4316-9939-f0977aeecde7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") "
Nov 03 16:27:47.588828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.587263    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-istio-kubeconfig\") pod \"ab567ff1-957a-4316-9939-f0977aeecde7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") "
Nov 03 16:27:47.588828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.587312    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-istio-csr-dns-cert\") pod \"ab567ff1-957a-4316-9939-f0977aeecde7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") "
Nov 03 16:27:47.588828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.587372    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab567ff1-957a-4316-9939-f0977aeecde7-kube-api-access-fvqz5\" (UniqueName: \"kubernetes.io/projected/ab567ff1-957a-4316-9939-f0977aeecde7-kube-api-access-fvqz5\") pod \"ab567ff1-957a-4316-9939-f0977aeecde7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") "
Nov 03 16:27:47.588828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.587419    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-cacerts\" (UniqueName: \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-cacerts\") pod \"ab567ff1-957a-4316-9939-f0977aeecde7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") "
Nov 03 16:27:47.588828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.587501    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab567ff1-957a-4316-9939-f0977aeecde7-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab567ff1-957a-4316-9939-f0977aeecde7-istio-csr-ca-configmap\") pod \"ab567ff1-957a-4316-9939-f0977aeecde7\" (UID: \"ab567ff1-957a-4316-9939-f0977aeecde7\") "
Nov 03 16:27:47.588828 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.588193    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab567ff1-957a-4316-9939-f0977aeecde7-istio-csr-ca-configmap" pod "ab567ff1-957a-4316-9939-f0977aeecde7" (UID: "ab567ff1-957a-4316-9939-f0977aeecde7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:27:47.593384 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.592936    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab567ff1-957a-4316-9939-f0977aeecde7-local-certs" pod "ab567ff1-957a-4316-9939-f0977aeecde7" (UID: "ab567ff1-957a-4316-9939-f0977aeecde7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:27:47.593687 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.593585    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-istio-csr-dns-cert" pod "ab567ff1-957a-4316-9939-f0977aeecde7" (UID: "ab567ff1-957a-4316-9939-f0977aeecde7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:27:47.595868 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.595773    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-istio-kubeconfig" pod "ab567ff1-957a-4316-9939-f0977aeecde7" (UID: "ab567ff1-957a-4316-9939-f0977aeecde7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:27:47.596203 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.596174    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-cacerts" pod "ab567ff1-957a-4316-9939-f0977aeecde7" (UID: "ab567ff1-957a-4316-9939-f0977aeecde7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:27:47.596813 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.596755    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab567ff1-957a-4316-9939-f0977aeecde7-kube-api-access-fvqz5" pod "ab567ff1-957a-4316-9939-f0977aeecde7" (UID: "ab567ff1-957a-4316-9939-f0977aeecde7"). InnerVolumeSpecName "kube-api-access-fvqz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:27:47.596908 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.596811    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab567ff1-957a-4316-9939-f0977aeecde7-istio-token" pod "ab567ff1-957a-4316-9939-f0977aeecde7" (UID: "ab567ff1-957a-4316-9939-f0977aeecde7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:27:47.688905 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.688816    6506 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-istio-kubeconfig\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:47.688905 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.688862    6506 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-istio-csr-dns-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:47.688905 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.688880    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvqz5\" (UniqueName: \"kubernetes.io/projected/ab567ff1-957a-4316-9939-f0977aeecde7-kube-api-access-fvqz5\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:47.688905 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.688897    6506 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab567ff1-957a-4316-9939-f0977aeecde7-cacerts\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:47.689149 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.688915    6506 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab567ff1-957a-4316-9939-f0977aeecde7-istio-csr-ca-configmap\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:47.689149 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.688929    6506 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab567ff1-957a-4316-9939-f0977aeecde7-local-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:47.689149 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:47.688943    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab567ff1-957a-4316-9939-f0977aeecde7-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:48.210818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:48.207642    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"
Nov 03 16:27:48.210818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:48.207845    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7" event={"ID":"ab567ff1-957a-4316-9939-f0977aeecde7","Type":"ContainerDied","Data":"8e09d8c164b0a04e9559091287f3702c6ec79b1e01329c7d3bd768ec2118868b"}
Nov 03 16:27:48.210818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:48.207907    6506 scope.go:122] "RemoveContainer" containerID="4c66698a2713b66534c4ef978011b746a08499f03bbcf22551930831cf29800a"
Nov 03 16:27:48.287182 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:48.287156    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"]
Nov 03 16:27:48.293482 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:48.293451    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trdp7"]
Nov 03 16:27:48.683962 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:48.683871    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"
Nov 03 16:27:48.703207 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:48.703181    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d917a62-fdad-458f-abcb-7267f36b02f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d917a62-fdad-458f-abcb-7267f36b02f7-maas-api-openshift-ca\") pod \"4d917a62-fdad-458f-abcb-7267f36b02f7\" (UID: \"4d917a62-fdad-458f-abcb-7267f36b02f7\") "
Nov 03 16:27:48.703428 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:48.703289    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d917a62-fdad-458f-abcb-7267f36b02f7-kube-api-access-887x2\" (UniqueName: \"kubernetes.io/projected/4d917a62-fdad-458f-abcb-7267f36b02f7-kube-api-access-887x2\") pod \"4d917a62-fdad-458f-abcb-7267f36b02f7\" (UID: \"4d917a62-fdad-458f-abcb-7267f36b02f7\") "
Nov 03 16:27:48.703428 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:48.703331    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d917a62-fdad-458f-abcb-7267f36b02f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d917a62-fdad-458f-abcb-7267f36b02f7-maas-api-xks-ca\") pod \"4d917a62-fdad-458f-abcb-7267f36b02f7\" (UID: \"4d917a62-fdad-458f-abcb-7267f36b02f7\") "
Nov 03 16:27:48.703561 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:48.703538    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d917a62-fdad-458f-abcb-7267f36b02f7-maas-api-openshift-ca" pod "4d917a62-fdad-458f-abcb-7267f36b02f7" (UID: "4d917a62-fdad-458f-abcb-7267f36b02f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:27:48.705660 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:48.705587    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d917a62-fdad-458f-abcb-7267f36b02f7-kube-api-access-887x2" pod "4d917a62-fdad-458f-abcb-7267f36b02f7" (UID: "4d917a62-fdad-458f-abcb-7267f36b02f7"). InnerVolumeSpecName "kube-api-access-887x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:27:48.705660 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:48.705589    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d917a62-fdad-458f-abcb-7267f36b02f7-maas-api-xks-ca" pod "4d917a62-fdad-458f-abcb-7267f36b02f7" (UID: "4d917a62-fdad-458f-abcb-7267f36b02f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:27:48.804680 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:48.804624    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d917a62-fdad-458f-abcb-7267f36b02f7-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:48.804680 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:48.804650    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887x2\" (UniqueName: \"kubernetes.io/projected/4d917a62-fdad-458f-abcb-7267f36b02f7-kube-api-access-887x2\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:48.804680 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:48.804661    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d917a62-fdad-458f-abcb-7267f36b02f7-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:27:49.214499 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:49.214469    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"
Nov 03 16:27:49.214499 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:49.214486    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96" event={"ID":"4d917a62-fdad-458f-abcb-7267f36b02f7","Type":"ContainerDied","Data":"1d451c84d274203d9980223840f9255cfb03867789896b077c7f85be680e9bde"}
Nov 03 16:27:49.215009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:49.214541    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d451c84d274203d9980223840f9255cfb03867789896b077c7f85be680e9bde"
Nov 03 16:27:50.102461 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:27:50.102425    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab567ff1-957a-4316-9939-f0977aeecde7" path="/var/lib/kubelet/pods/ab567ff1-957a-4316-9939-f0977aeecde7/volumes"
Nov 03 16:27:58.796114 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:27:58.796075    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:28:04.266690 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:04.266654    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:28:06.618723 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:06.618689    6506 scope.go:122] "RemoveContainer" containerID="9e1a174b6fa72b59cddbab456112bca984ca67b94b65670c1c6d7a6fb153f8e1"
Nov 03 16:28:06.627623 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:06.627597    6506 scope.go:122] "RemoveContainer" containerID="d972433f8be6db243ebd133cd8443363647a8b4ea77de8340f8c329d745d2c43"
Nov 03 16:28:06.635636 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:06.635618    6506 scope.go:122] "RemoveContainer" containerID="c17d1e150c08262e3e01fbf864b297d069c7e6b28103a458d6fbc86a4f80fed5"
Nov 03 16:28:06.643319 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:06.643301    6506 scope.go:122] "RemoveContainer" containerID="e6c2a3e3780191ca8989710c1eeae264827b5ececc471118eb27bb73a66b252c"
Nov 03 16:28:06.650870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:06.650856    6506 scope.go:122] "RemoveContainer" containerID="c265582b80f2804620a220b3b026ff703abcd793cc14ca8ad578c0f5b36b83fd"
Nov 03 16:28:06.658548 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:06.658530    6506 scope.go:122] "RemoveContainer" containerID="6d2742712fdaf44f345434f2a5f499d9a12352c595a1124ed797df0a198b427d"
Nov 03 16:28:06.666102 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:06.666085    6506 scope.go:122] "RemoveContainer" containerID="8fd1a3569b5d68968e07a737e9a56d8c114bcb8e8f7e039f8e9e25e6f7825c30"
Nov 03 16:28:06.673675 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:06.673655    6506 scope.go:122] "RemoveContainer" containerID="b7aa383af6563f09b3ae8d4c306f21d0a5e80b06d43b673fb72d672f4b98a4d0"
Nov 03 16:28:06.680998 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:06.680980    6506 scope.go:122] "RemoveContainer" containerID="75d32a1a94debd0cdc1a39ab16e9c495e1a3eb9480d708b6bf3d11c5ca1220af"
Nov 03 16:28:06.691257 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:06.691222    6506 scope.go:122] "RemoveContainer" containerID="83c6cb44d3a308f9f7e46f6f519f3cfc4eb9071e5843c3cca8426fc6db13f3ce"
Nov 03 16:28:10.065195 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:10.065161    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:28:10.243937 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:10.243902    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:28:11.133061 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:11.133028    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:28:15.084776 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:15.084743    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:28:17.024870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:17.024837    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"]
Nov 03 16:28:17.029505 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:17.029480    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2727v"]
Nov 03 16:28:18.101909 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:28:18.101875    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17" path="/var/lib/kubelet/pods/7fa8e4f0-6e43-4cd9-9cb2-bfbd55627b17/volumes"
Nov 03 16:28:28.795632 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:28:28.795590    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:28:58.795972 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:28:58.795891    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:29:06.749467 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:29:06.749440    6506 scope.go:122] "RemoveContainer" containerID="c5fdfc94540a8f62b34a3e2196b3726b8deb92735288a2e1d5785abf888bc118"
Nov 03 16:29:06.758338 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:29:06.758321    6506 scope.go:122] "RemoveContainer" containerID="8603f3f07726846ff08ccfa82f369ad3277292a8fc5d0e98f4cb3e32dabf0a6c"
Nov 03 16:29:10.116915 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:29:10.116874    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:29:10.275609 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:29:10.275566    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:29:13.023522 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:29:13.023489    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"]
Nov 03 16:29:13.026436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:29:13.026411    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cq7cd"]
Nov 03 16:29:14.101694 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:29:14.101659    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c033b89f-0694-4636-b45c-468c655c2e3c" path="/var/lib/kubelet/pods/c033b89f-0694-4636-b45c-468c655c2e3c/volumes"
Nov 03 16:29:28.795700 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:29:28.795662    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:29:34.047208 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:29:34.047173    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"]
Nov 03 16:29:34.049177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:29:34.049146    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6h4d"]
Nov 03 16:29:34.102285 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:29:34.102255    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58acba74-0ea8-41a2-8bc6-3c15c3257e21" path="/var/lib/kubelet/pods/58acba74-0ea8-41a2-8bc6-3c15c3257e21/volumes"
Nov 03 16:29:45.025214 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:29:45.025177    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"]
Nov 03 16:29:45.027803 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:29:45.027769    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qh9n8"]
Nov 03 16:29:46.102389 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:29:46.102355    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c472bb55-e437-4583-a52f-dc2ec9d8b4ca" path="/var/lib/kubelet/pods/c472bb55-e437-4583-a52f-dc2ec9d8b4ca/volumes"
Nov 03 16:29:50.468572 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:29:50.468533    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:29:58.796162 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:29:58.796127    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:30:00.157746 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.157714    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"]
Nov 03 16:30:00.158150 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.158096    6506 cpu_manager.go:414] "containerMap: removing container" podUID="ab567ff1-957a-4316-9939-f0977aeecde7" containerName="discovery"
Nov 03 16:30:00.158150 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.158108    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab567ff1-957a-4316-9939-f0977aeecde7" containerName="discovery"
Nov 03 16:30:00.158150 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.158151    6506 cpu_manager.go:414] "containerMap: removing container" podUID="4d917a62-fdad-458f-abcb-7267f36b02f7" containerName="revoke-keys"
Nov 03 16:30:00.158270 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.158157    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d917a62-fdad-458f-abcb-7267f36b02f7" containerName="revoke-keys"
Nov 03 16:30:00.158270 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.158231    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d917a62-fdad-458f-abcb-7267f36b02f7" containerName="revoke-keys"
Nov 03 16:30:00.158270 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.158241    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab567ff1-957a-4316-9939-f0977aeecde7" containerName="discovery"
Nov 03 16:30:00.160804 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.160779    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"
Nov 03 16:30:00.166213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.166190    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:30:00.166213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.166201    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:30:00.166412 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.166226    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:30:00.193204 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.193165    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"]
Nov 03 16:30:00.204308 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.204280    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87fe089d-40b1-479b-99bb-ed8940822054-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895390-v4q9l\" (UID: \"87fe089d-40b1-479b-99bb-ed8940822054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"
Nov 03 16:30:00.204434 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.204411    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j5qg\" (UniqueName: \"kubernetes.io/projected/87fe089d-40b1-479b-99bb-ed8940822054-kube-api-access-4j5qg\") pod \"maas-api-key-cleanup-29895390-v4q9l\" (UID: \"87fe089d-40b1-479b-99bb-ed8940822054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"
Nov 03 16:30:00.305604 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.305573    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87fe089d-40b1-479b-99bb-ed8940822054-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895390-v4q9l\" (UID: \"87fe089d-40b1-479b-99bb-ed8940822054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"
Nov 03 16:30:00.305738 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.305672    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j5qg\" (UniqueName: \"kubernetes.io/projected/87fe089d-40b1-479b-99bb-ed8940822054-kube-api-access-4j5qg\") pod \"maas-api-key-cleanup-29895390-v4q9l\" (UID: \"87fe089d-40b1-479b-99bb-ed8940822054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"
Nov 03 16:30:00.306269 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.306251    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87fe089d-40b1-479b-99bb-ed8940822054-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895390-v4q9l\" (UID: \"87fe089d-40b1-479b-99bb-ed8940822054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"
Nov 03 16:30:00.319069 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.319041    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j5qg\" (UniqueName: \"kubernetes.io/projected/87fe089d-40b1-479b-99bb-ed8940822054-kube-api-access-4j5qg\") pod \"maas-api-key-cleanup-29895390-v4q9l\" (UID: \"87fe089d-40b1-479b-99bb-ed8940822054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"
Nov 03 16:30:00.473865 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.473777    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"
Nov 03 16:30:00.750395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.750368    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"]
Nov 03 16:30:00.752247 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:30:00.752216    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87fe089d_40b1_479b_99bb_ed8940822054.slice/crio-76d49653426074315d8beb1e8af6814ed4973628ae0654954f2241fa50aa6a65 WatchSource:0}: Error finding container 76d49653426074315d8beb1e8af6814ed4973628ae0654954f2241fa50aa6a65: Status 404 returned error can't find the container with id 76d49653426074315d8beb1e8af6814ed4973628ae0654954f2241fa50aa6a65
Nov 03 16:30:00.960809 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.960762    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l" event={"ID":"87fe089d-40b1-479b-99bb-ed8940822054","Type":"ContainerStarted","Data":"28438771decad7ba3e33163e41ae954ae854b127bc10228c80fcb3ab04322e13"}
Nov 03 16:30:00.960918 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.960840    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l" event={"ID":"87fe089d-40b1-479b-99bb-ed8940822054","Type":"ContainerStarted","Data":"76d49653426074315d8beb1e8af6814ed4973628ae0654954f2241fa50aa6a65"}
Nov 03 16:30:00.980723 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:00.980675    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l" podStartSLOduration=0.980661255 podStartE2EDuration="980.661255ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:00.979341455 +0000 UTC m=+1614.992739599" watchObservedRunningTime="2026-11-03 16:30:00.980661255 +0000 UTC m=+1614.994059401"
Nov 03 16:30:01.967682 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:01.967593    6506 generic.go:356] "Generic (PLEG): container finished" podID="87fe089d-40b1-479b-99bb-ed8940822054" containerID="28438771decad7ba3e33163e41ae954ae854b127bc10228c80fcb3ab04322e13" exitCode=60
Nov 03 16:30:01.968110 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:01.967675    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l" event={"ID":"87fe089d-40b1-479b-99bb-ed8940822054","Type":"ContainerDied","Data":"28438771decad7ba3e33163e41ae954ae854b127bc10228c80fcb3ab04322e13"}
Nov 03 16:30:01.968110 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:01.967964    6506 scope.go:122] "RemoveContainer" containerID="28438771decad7ba3e33163e41ae954ae854b127bc10228c80fcb3ab04322e13"
Nov 03 16:30:02.975288 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:02.975255    6506 generic.go:356] "Generic (PLEG): container finished" podID="87fe089d-40b1-479b-99bb-ed8940822054" containerID="d52b247a78ae9461c205de204a858d15208030665703d0afa2633eb6e5a9bee4" exitCode=60
Nov 03 16:30:02.975737 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:02.975345    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l" event={"ID":"87fe089d-40b1-479b-99bb-ed8940822054","Type":"ContainerDied","Data":"d52b247a78ae9461c205de204a858d15208030665703d0afa2633eb6e5a9bee4"}
Nov 03 16:30:02.975737 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:02.975409    6506 scope.go:122] "RemoveContainer" containerID="28438771decad7ba3e33163e41ae954ae854b127bc10228c80fcb3ab04322e13"
Nov 03 16:30:02.975737 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:02.975600    6506 scope.go:122] "RemoveContainer" containerID="d52b247a78ae9461c205de204a858d15208030665703d0afa2633eb6e5a9bee4"
Nov 03 16:30:02.975897 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:30:02.975880    6506 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895390-v4q9l_odh-ai-gateway-infra(87fe089d-40b1-479b-99bb-ed8940822054)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l" podUID="87fe089d-40b1-479b-99bb-ed8940822054"
Nov 03 16:30:03.982641 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:03.982612    6506 scope.go:122] "RemoveContainer" containerID="d52b247a78ae9461c205de204a858d15208030665703d0afa2633eb6e5a9bee4"
Nov 03 16:30:03.983120 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:30:03.982925    6506 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895390-v4q9l_odh-ai-gateway-infra(87fe089d-40b1-479b-99bb-ed8940822054)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l" podUID="87fe089d-40b1-479b-99bb-ed8940822054"
Nov 03 16:30:06.783299 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:06.783265    6506 scope.go:122] "RemoveContainer" containerID="05ba2a48255e28ee4a90216d8521e94be693db2fa0e4d5335e8417b1670c6abe"
Nov 03 16:30:10.170312 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:10.170261    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:30:11.024937 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:11.024902    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"]
Nov 03 16:30:11.027967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:11.027946    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qd277"]
Nov 03 16:30:12.102329 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:12.102287    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248fadfb-54a8-4b6c-a13b-9c99b2a25b74" path="/var/lib/kubelet/pods/248fadfb-54a8-4b6c-a13b-9c99b2a25b74/volumes"
Nov 03 16:30:16.099417 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:16.099330    6506 scope.go:122] "RemoveContainer" containerID="d52b247a78ae9461c205de204a858d15208030665703d0afa2633eb6e5a9bee4"
Nov 03 16:30:17.053052 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:17.053018    6506 generic.go:356] "Generic (PLEG): container finished" podID="87fe089d-40b1-479b-99bb-ed8940822054" containerID="5a9042c92fa2f3e39b5279294de124d6cc3e68cfbdf05ea6066b58268645a73d" exitCode=60
Nov 03 16:30:17.053230 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:17.053090    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l" event={"ID":"87fe089d-40b1-479b-99bb-ed8940822054","Type":"ContainerDied","Data":"5a9042c92fa2f3e39b5279294de124d6cc3e68cfbdf05ea6066b58268645a73d"}
Nov 03 16:30:17.053230 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:17.053153    6506 scope.go:122] "RemoveContainer" containerID="d52b247a78ae9461c205de204a858d15208030665703d0afa2633eb6e5a9bee4"
Nov 03 16:30:17.053563 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:17.053541    6506 scope.go:122] "RemoveContainer" containerID="5a9042c92fa2f3e39b5279294de124d6cc3e68cfbdf05ea6066b58268645a73d"
Nov 03 16:30:17.053833 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:30:17.053812    6506 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895390-v4q9l_odh-ai-gateway-infra(87fe089d-40b1-479b-99bb-ed8940822054)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l" podUID="87fe089d-40b1-479b-99bb-ed8940822054"
Nov 03 16:30:18.112022 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:18.111993    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"]
Nov 03 16:30:18.250407 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:18.250381    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"
Nov 03 16:30:18.379721 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:18.379636    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87fe089d-40b1-479b-99bb-ed8940822054-kube-api-access-4j5qg\" (UniqueName: \"kubernetes.io/projected/87fe089d-40b1-479b-99bb-ed8940822054-kube-api-access-4j5qg\") pod \"87fe089d-40b1-479b-99bb-ed8940822054\" (UID: \"87fe089d-40b1-479b-99bb-ed8940822054\") "
Nov 03 16:30:18.379894 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:18.379732    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87fe089d-40b1-479b-99bb-ed8940822054-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87fe089d-40b1-479b-99bb-ed8940822054-maas-api-service-ca\") pod \"87fe089d-40b1-479b-99bb-ed8940822054\" (UID: \"87fe089d-40b1-479b-99bb-ed8940822054\") "
Nov 03 16:30:18.380097 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:18.380074    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87fe089d-40b1-479b-99bb-ed8940822054-maas-api-service-ca" pod "87fe089d-40b1-479b-99bb-ed8940822054" (UID: "87fe089d-40b1-479b-99bb-ed8940822054"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:30:18.381967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:18.381950    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87fe089d-40b1-479b-99bb-ed8940822054-kube-api-access-4j5qg" pod "87fe089d-40b1-479b-99bb-ed8940822054" (UID: "87fe089d-40b1-479b-99bb-ed8940822054"). InnerVolumeSpecName "kube-api-access-4j5qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:30:18.480517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:18.480482    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87fe089d-40b1-479b-99bb-ed8940822054-maas-api-service-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:30:18.480517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:18.480512    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j5qg\" (UniqueName: \"kubernetes.io/projected/87fe089d-40b1-479b-99bb-ed8940822054-kube-api-access-4j5qg\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:30:19.066224 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:19.066197    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"
Nov 03 16:30:19.066411 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:19.066227    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l" event={"ID":"87fe089d-40b1-479b-99bb-ed8940822054","Type":"ContainerDied","Data":"76d49653426074315d8beb1e8af6814ed4973628ae0654954f2241fa50aa6a65"}
Nov 03 16:30:19.066411 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:19.066286    6506 scope.go:122] "RemoveContainer" containerID="5a9042c92fa2f3e39b5279294de124d6cc3e68cfbdf05ea6066b58268645a73d"
Nov 03 16:30:19.089030 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:19.089008    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"]
Nov 03 16:30:19.093219 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:19.093196    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895390-v4q9l"]
Nov 03 16:30:20.101969 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:20.101937    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87fe089d-40b1-479b-99bb-ed8940822054" path="/var/lib/kubelet/pods/87fe089d-40b1-479b-99bb-ed8940822054/volumes"
Nov 03 16:30:22.022851 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:22.022788    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"]
Nov 03 16:30:22.024854 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:22.024828    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5r6m6"]
Nov 03 16:30:22.102217 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:22.102184    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c39a05-9d40-489a-b0ee-c65f394ad8ff" path="/var/lib/kubelet/pods/48c39a05-9d40-489a-b0ee-c65f394ad8ff/volumes"
Nov 03 16:30:28.796659 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:30:28.796620    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:30:42.022327 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:42.022282    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"]
Nov 03 16:30:42.025240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:42.025220    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xzb9p"]
Nov 03 16:30:42.101734 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:30:42.101704    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b5fa86-c589-429f-bb0f-2a5f2c7ec248" path="/var/lib/kubelet/pods/d7b5fa86-c589-429f-bb0f-2a5f2c7ec248/volumes"
Nov 03 16:30:58.796223 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:30:58.796177    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:31:00.023789 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:31:00.023754    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"]
Nov 03 16:31:00.026334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:31:00.026307    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rkbb"]
Nov 03 16:31:00.104714 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:31:00.104680    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c11a05bb-37ae-45ba-b257-7553097de947" path="/var/lib/kubelet/pods/c11a05bb-37ae-45ba-b257-7553097de947/volumes"
Nov 03 16:31:06.806161 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:31:06.806129    6506 scope.go:122] "RemoveContainer" containerID="4d2727a685d1da6b585950fd7c0b90eaef79d95b09240e1d4267250ccd2ed60e"
Nov 03 16:31:06.816085 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:31:06.816067    6506 scope.go:122] "RemoveContainer" containerID="481af62c6cf326bdaca148480498960d325ae33783c6cbb4409d8f6b08dea912"
Nov 03 16:31:06.824391 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:31:06.824370    6506 scope.go:122] "RemoveContainer" containerID="b532841c533e68ae56b46e17bb56671161fb92621078814137199cdcfded1c9e"
Nov 03 16:31:10.224412 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:31:10.224354    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:31:19.025309 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:31:19.025270    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"]
Nov 03 16:31:19.028829 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:31:19.028788    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dzlvl"]
Nov 03 16:31:20.102209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:31:20.102176    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06bcbbde-9851-4747-8807-1089b1bb57c3" path="/var/lib/kubelet/pods/06bcbbde-9851-4747-8807-1089b1bb57c3/volumes"
Nov 03 16:31:28.796335 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:31:28.796297    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:31:37.035304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:31:37.035268    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"]
Nov 03 16:31:37.040002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:31:37.039980    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gvwq6"]
Nov 03 16:31:38.101835 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:31:38.101779    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1cbba90-1807-412b-b826-1537db115c28" path="/var/lib/kubelet/pods/c1cbba90-1807-412b-b826-1537db115c28/volumes"
Nov 03 16:31:58.796037 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:31:58.796000    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:32:06.856176 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:32:06.856143    6506 scope.go:122] "RemoveContainer" containerID="425216545d382bc9f5e92b23e86f48f6d8e5137f8c3402d022c9ef2a21236d98"
Nov 03 16:32:06.865395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:32:06.865375    6506 scope.go:122] "RemoveContainer" containerID="17a760adf53390bcf4b0886db7dda72feccdaccba04dfcb7b826a235931f94fa"
Nov 03 16:32:06.875159 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:32:06.875134    6506 scope.go:122] "RemoveContainer" containerID="b53515f28e07966dc6d67a7df38657f0743cb08fff127f3ce1ab267c6fb87adc"
Nov 03 16:32:06.883275 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:32:06.883252    6506 scope.go:122] "RemoveContainer" containerID="2e4d7a34be0f22969545404db89f94cb0fbd0073cc3220c549c767ee7e719b7a"
Nov 03 16:32:10.024544 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:32:10.024514    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"]
Nov 03 16:32:10.027364 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:32:10.027343    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptr25"]
Nov 03 16:32:10.102373 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:32:10.102339    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd555c7-745a-4cf4-9bae-13e2793ff62b" path="/var/lib/kubelet/pods/3bd555c7-745a-4cf4-9bae-13e2793ff62b/volumes"
Nov 03 16:32:10.279951 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:32:10.279817    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:32:28.796148 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:32:28.796114    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:32:39.022091 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:32:39.022055    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"]
Nov 03 16:32:39.027084 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:32:39.027057    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xv69m"]
Nov 03 16:32:40.102378 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:32:40.102345    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fdb6dd9-c8da-430e-8571-a8bd31214c2f" path="/var/lib/kubelet/pods/5fdb6dd9-c8da-430e-8571-a8bd31214c2f/volumes"
Nov 03 16:32:49.025984 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:32:49.025946    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"]
Nov 03 16:32:49.028908 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:32:49.028883    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7mf96"]
Nov 03 16:32:50.102341 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:32:50.102304    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d917a62-fdad-458f-abcb-7267f36b02f7" path="/var/lib/kubelet/pods/4d917a62-fdad-458f-abcb-7267f36b02f7/volumes"
Nov 03 16:32:58.795704 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:32:58.795667    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:33:01.054994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:01.054949    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:04.191262 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.191226    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"]
Nov 03 16:33:04.191746 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.191605    6506 cpu_manager.go:414] "containerMap: removing container" podUID="87fe089d-40b1-479b-99bb-ed8940822054" containerName="cleanup"
Nov 03 16:33:04.191746 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.191617    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87fe089d-40b1-479b-99bb-ed8940822054" containerName="cleanup"
Nov 03 16:33:04.191746 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.191633    6506 cpu_manager.go:414] "containerMap: removing container" podUID="87fe089d-40b1-479b-99bb-ed8940822054" containerName="cleanup"
Nov 03 16:33:04.191746 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.191638    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87fe089d-40b1-479b-99bb-ed8940822054" containerName="cleanup"
Nov 03 16:33:04.191996 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.191745    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="87fe089d-40b1-479b-99bb-ed8940822054" containerName="cleanup"
Nov 03 16:33:04.191996 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.191762    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="87fe089d-40b1-479b-99bb-ed8940822054" containerName="cleanup"
Nov 03 16:33:04.194932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.194916    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"
Nov 03 16:33:04.200518 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.200499    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:33:04.200623 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.200515    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:33:04.218479 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.218451    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"]
Nov 03 16:33:04.229273 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.229248    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5144d016-0056-4b27-b448-6eadf8287a7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rf5xw\" (UID: \"5144d016-0056-4b27-b448-6eadf8287a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"
Nov 03 16:33:04.229400 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.229286    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtj2x\" (UniqueName: \"kubernetes.io/projected/5144d016-0056-4b27-b448-6eadf8287a7a-kube-api-access-jtj2x\") pod \"maas-api-revoke-sub-3d4156f1-rf5xw\" (UID: \"5144d016-0056-4b27-b448-6eadf8287a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"
Nov 03 16:33:04.229400 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.229318    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5144d016-0056-4b27-b448-6eadf8287a7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rf5xw\" (UID: \"5144d016-0056-4b27-b448-6eadf8287a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"
Nov 03 16:33:04.252478 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.252450    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:33:04.252584 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.252504    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:33:04.329894 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.329863    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5144d016-0056-4b27-b448-6eadf8287a7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rf5xw\" (UID: \"5144d016-0056-4b27-b448-6eadf8287a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"
Nov 03 16:33:04.329894 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.329897    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtj2x\" (UniqueName: \"kubernetes.io/projected/5144d016-0056-4b27-b448-6eadf8287a7a-kube-api-access-jtj2x\") pod \"maas-api-revoke-sub-3d4156f1-rf5xw\" (UID: \"5144d016-0056-4b27-b448-6eadf8287a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"
Nov 03 16:33:04.330156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.329919    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5144d016-0056-4b27-b448-6eadf8287a7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rf5xw\" (UID: \"5144d016-0056-4b27-b448-6eadf8287a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"
Nov 03 16:33:04.330826 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.330692    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5144d016-0056-4b27-b448-6eadf8287a7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rf5xw\" (UID: \"5144d016-0056-4b27-b448-6eadf8287a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"
Nov 03 16:33:04.332417 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.332395    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5144d016-0056-4b27-b448-6eadf8287a7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rf5xw\" (UID: \"5144d016-0056-4b27-b448-6eadf8287a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"
Nov 03 16:33:04.342389 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.342364    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtj2x\" (UniqueName: \"kubernetes.io/projected/5144d016-0056-4b27-b448-6eadf8287a7a-kube-api-access-jtj2x\") pod \"maas-api-revoke-sub-3d4156f1-rf5xw\" (UID: \"5144d016-0056-4b27-b448-6eadf8287a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"
Nov 03 16:33:04.507320 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.507238    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"
Nov 03 16:33:04.782343 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.782314    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"]
Nov 03 16:33:04.784440 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:33:04.784415    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5144d016_0056_4b27_b448_6eadf8287a7a.slice/crio-a4f14e8c749487448496a964ff5e2bc23468e6961a039021ce26f094df1e5fde WatchSource:0}: Error finding container a4f14e8c749487448496a964ff5e2bc23468e6961a039021ce26f094df1e5fde: Status 404 returned error can't find the container with id a4f14e8c749487448496a964ff5e2bc23468e6961a039021ce26f094df1e5fde
Nov 03 16:33:04.945655 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:04.945620    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw" event={"ID":"5144d016-0056-4b27-b448-6eadf8287a7a","Type":"ContainerStarted","Data":"a4f14e8c749487448496a964ff5e2bc23468e6961a039021ce26f094df1e5fde"}
Nov 03 16:33:05.952495 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:05.952456    6506 generic.go:356] "Generic (PLEG): container finished" podID="5144d016-0056-4b27-b448-6eadf8287a7a" containerID="5d3306cc68eb533b98375dfc1a5098681764716558b3260c8fe6822e6fb8ffa2" exitCode=0
Nov 03 16:33:05.952938 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:05.952536    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw" event={"ID":"5144d016-0056-4b27-b448-6eadf8287a7a","Type":"ContainerDied","Data":"5d3306cc68eb533b98375dfc1a5098681764716558b3260c8fe6822e6fb8ffa2"}
Nov 03 16:33:06.914161 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:06.914127    6506 scope.go:122] "RemoveContainer" containerID="9b6de4122ddf49897ead903cb8a58ef8154ee2102825fc1a4b36c2fcd4e65a36"
Nov 03 16:33:06.925663 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:06.925634    6506 scope.go:122] "RemoveContainer" containerID="41415ea4bfb89dd1ab53e8207143875a235d621905bc1adc01192c50d3be1813"
Nov 03 16:33:06.934299 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:06.934279    6506 scope.go:122] "RemoveContainer" containerID="154c5d4c9abc26a10779548b367f1a0f5a303e310f2913ec58a59e15bd98ac9a"
Nov 03 16:33:06.942234 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:06.942217    6506 scope.go:122] "RemoveContainer" containerID="2ee3b90beb1f7a1cdf98c13ea6ba7204a591b0cfcf1d287a662e61aa2a1be5c7"
Nov 03 16:33:07.098903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:07.098874    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"
Nov 03 16:33:07.156537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:07.156501    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5144d016-0056-4b27-b448-6eadf8287a7a-kube-api-access-jtj2x\" (UniqueName: \"kubernetes.io/projected/5144d016-0056-4b27-b448-6eadf8287a7a-kube-api-access-jtj2x\") pod \"5144d016-0056-4b27-b448-6eadf8287a7a\" (UID: \"5144d016-0056-4b27-b448-6eadf8287a7a\") "
Nov 03 16:33:07.156710 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:07.156630    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5144d016-0056-4b27-b448-6eadf8287a7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5144d016-0056-4b27-b448-6eadf8287a7a-maas-api-openshift-ca\") pod \"5144d016-0056-4b27-b448-6eadf8287a7a\" (UID: \"5144d016-0056-4b27-b448-6eadf8287a7a\") "
Nov 03 16:33:07.156710 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:07.156664    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5144d016-0056-4b27-b448-6eadf8287a7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5144d016-0056-4b27-b448-6eadf8287a7a-maas-api-xks-ca\") pod \"5144d016-0056-4b27-b448-6eadf8287a7a\" (UID: \"5144d016-0056-4b27-b448-6eadf8287a7a\") "
Nov 03 16:33:07.157067 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:07.157039    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5144d016-0056-4b27-b448-6eadf8287a7a-maas-api-openshift-ca" pod "5144d016-0056-4b27-b448-6eadf8287a7a" (UID: "5144d016-0056-4b27-b448-6eadf8287a7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:33:07.158826 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:07.158784    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5144d016-0056-4b27-b448-6eadf8287a7a-maas-api-xks-ca" pod "5144d016-0056-4b27-b448-6eadf8287a7a" (UID: "5144d016-0056-4b27-b448-6eadf8287a7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:33:07.158907 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:07.158849    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5144d016-0056-4b27-b448-6eadf8287a7a-kube-api-access-jtj2x" pod "5144d016-0056-4b27-b448-6eadf8287a7a" (UID: "5144d016-0056-4b27-b448-6eadf8287a7a"). InnerVolumeSpecName "kube-api-access-jtj2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:33:07.258119 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:07.258076    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5144d016-0056-4b27-b448-6eadf8287a7a-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:07.258119 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:07.258108    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5144d016-0056-4b27-b448-6eadf8287a7a-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:07.258119 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:07.258119    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtj2x\" (UniqueName: \"kubernetes.io/projected/5144d016-0056-4b27-b448-6eadf8287a7a-kube-api-access-jtj2x\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:07.800756 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:07.800712    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"]
Nov 03 16:33:07.801035 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:07.800990    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" podUID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" containerName="istio-proxy" containerID="cri-o://197f23e8960b82eae5097ce89bc5f2503f05db4727e6484631cdc6cdd358ad08" gracePeriod=30
Nov 03 16:33:07.967410 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:07.967377    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"
Nov 03 16:33:07.967410 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:07.967389    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw" event={"ID":"5144d016-0056-4b27-b448-6eadf8287a7a","Type":"ContainerDied","Data":"a4f14e8c749487448496a964ff5e2bc23468e6961a039021ce26f094df1e5fde"}
Nov 03 16:33:07.967613 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:07.967457    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4f14e8c749487448496a964ff5e2bc23468e6961a039021ce26f094df1e5fde"
Nov 03 16:33:10.277624 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:10.277589    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:12.995323 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:12.995290    6506 generic.go:356] "Generic (PLEG): container finished" podID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" containerID="197f23e8960b82eae5097ce89bc5f2503f05db4727e6484631cdc6cdd358ad08" exitCode=0
Nov 03 16:33:12.995706 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:12.995336    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" event={"ID":"01649a37-cda3-4c59-8bd5-a3a1771d6fcc","Type":"ContainerDied","Data":"197f23e8960b82eae5097ce89bc5f2503f05db4727e6484631cdc6cdd358ad08"}
Nov 03 16:33:13.040218 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.040193    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:33:13.112550 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.112474    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-podinfo\") pod \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") "
Nov 03 16:33:13.112550 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.112518    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-data\") pod \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") "
Nov 03 16:33:13.112780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.112555    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istiod-ca-cert\") pod \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") "
Nov 03 16:33:13.112780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.112665    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-envoy\") pod \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") "
Nov 03 16:33:13.113118 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.113066    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istiod-ca-cert" pod "01649a37-cda3-4c59-8bd5-a3a1771d6fcc" (UID: "01649a37-cda3-4c59-8bd5-a3a1771d6fcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:33:13.113228 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.113101    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-kube-api-access-65nj4\" (UniqueName: \"kubernetes.io/projected/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-kube-api-access-65nj4\") pod \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") "
Nov 03 16:33:13.113228 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.113175    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-workload-socket\") pod \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") "
Nov 03 16:33:13.113353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.113241    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-token\" (UniqueName: \"kubernetes.io/projected/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-token\") pod \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") "
Nov 03 16:33:13.113353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.113279    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-credential-socket\") pod \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") "
Nov 03 16:33:13.113353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.113310    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-workload-certs\") pod \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\" (UID: \"01649a37-cda3-4c59-8bd5-a3a1771d6fcc\") "
Nov 03 16:33:13.113679 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.113597    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.113982 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.113809    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-data" pod "01649a37-cda3-4c59-8bd5-a3a1771d6fcc" (UID: "01649a37-cda3-4c59-8bd5-a3a1771d6fcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:13.114078 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.113981    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-workload-socket" pod "01649a37-cda3-4c59-8bd5-a3a1771d6fcc" (UID: "01649a37-cda3-4c59-8bd5-a3a1771d6fcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:13.114156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.114132    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-workload-certs" pod "01649a37-cda3-4c59-8bd5-a3a1771d6fcc" (UID: "01649a37-cda3-4c59-8bd5-a3a1771d6fcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:13.114268 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.114217    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-credential-socket" pod "01649a37-cda3-4c59-8bd5-a3a1771d6fcc" (UID: "01649a37-cda3-4c59-8bd5-a3a1771d6fcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:13.116141 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.116118    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-podinfo" pod "01649a37-cda3-4c59-8bd5-a3a1771d6fcc" (UID: "01649a37-cda3-4c59-8bd5-a3a1771d6fcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:33:13.116241 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.116139    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-kube-api-access-65nj4" pod "01649a37-cda3-4c59-8bd5-a3a1771d6fcc" (UID: "01649a37-cda3-4c59-8bd5-a3a1771d6fcc"). InnerVolumeSpecName "kube-api-access-65nj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:33:13.116480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.116457    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-envoy" pod "01649a37-cda3-4c59-8bd5-a3a1771d6fcc" (UID: "01649a37-cda3-4c59-8bd5-a3a1771d6fcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:13.117025 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.117007    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-token" pod "01649a37-cda3-4c59-8bd5-a3a1771d6fcc" (UID: "01649a37-cda3-4c59-8bd5-a3a1771d6fcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:33:13.215188 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.215158    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65nj4\" (UniqueName: \"kubernetes.io/projected/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-kube-api-access-65nj4\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.215188 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.215189    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.215395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.215200    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.215395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.215211    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.215395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.215221    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.215395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.215229    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.215395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.215238    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.215395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:13.215246    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01649a37-cda3-4c59-8bd5-a3a1771d6fcc-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:14.002287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:14.002254    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"
Nov 03 16:33:14.002760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:14.002280    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7" event={"ID":"01649a37-cda3-4c59-8bd5-a3a1771d6fcc","Type":"ContainerDied","Data":"dac9caecf09e81367d723f041e88c562756a4c32e94bd9c292d0016edaaaa280"}
Nov 03 16:33:14.002760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:14.002357    6506 scope.go:122] "RemoveContainer" containerID="197f23e8960b82eae5097ce89bc5f2503f05db4727e6484631cdc6cdd358ad08"
Nov 03 16:33:14.032521 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:14.032490    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"]
Nov 03 16:33:14.035702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:14.035676    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2prf7"]
Nov 03 16:33:14.102101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:14.102072    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" path="/var/lib/kubelet/pods/01649a37-cda3-4c59-8bd5-a3a1771d6fcc/volumes"
Nov 03 16:33:20.430081 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:20.430000    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:22.570345 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:22.570315    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:22.838152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:22.838073    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"]
Nov 03 16:33:22.838317 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:22.838295    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" podUID="13e172cb-f664-4da0-a922-08b5a0eb29d6" containerName="istio-proxy" containerID="cri-o://2ca18f7e3ca5b61c9d444d0e44c38cbade52bd818c7123d2e0fec5f33688cac6" gracePeriod=30
Nov 03 16:33:22.845366 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:22.845337    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"]
Nov 03 16:33:22.845576 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:22.845534    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerName="istio-proxy" containerID="cri-o://bc4429adad9428ed1eb2bda8566b67a254ccbd7aca5a1350e1ba74c096679a3c" gracePeriod=30
Nov 03 16:33:23.064085 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:23.064050    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:23.240206 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:23.240117    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:26.552649 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:26.552610    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:26.746274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:26.746245    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:27.720344 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.720315    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"]
Nov 03 16:33:27.720738 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.720682    6506 cpu_manager.go:414] "containerMap: removing container" podUID="87fe089d-40b1-479b-99bb-ed8940822054" containerName="cleanup"
Nov 03 16:33:27.720738 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.720694    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87fe089d-40b1-479b-99bb-ed8940822054" containerName="cleanup"
Nov 03 16:33:27.720738 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.720715    6506 cpu_manager.go:414] "containerMap: removing container" podUID="5144d016-0056-4b27-b448-6eadf8287a7a" containerName="revoke-keys"
Nov 03 16:33:27.720738 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.720720    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5144d016-0056-4b27-b448-6eadf8287a7a" containerName="revoke-keys"
Nov 03 16:33:27.720738 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.720739    6506 cpu_manager.go:414] "containerMap: removing container" podUID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" containerName="istio-proxy"
Nov 03 16:33:27.720976 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.720744    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" containerName="istio-proxy"
Nov 03 16:33:27.720976 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.720852    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="87fe089d-40b1-479b-99bb-ed8940822054" containerName="cleanup"
Nov 03 16:33:27.720976 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.720868    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="5144d016-0056-4b27-b448-6eadf8287a7a" containerName="revoke-keys"
Nov 03 16:33:27.720976 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.720879    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="01649a37-cda3-4c59-8bd5-a3a1771d6fcc" containerName="istio-proxy"
Nov 03 16:33:27.724100 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.724074    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:27.729851 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.729828    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:33:27.730174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.730152    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 16:33:27.732858 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.732834    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ba8f54-serving-cert\""
Nov 03 16:33:27.732858 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.732843    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:33:27.733027 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.732979    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v65q5\""
Nov 03 16:33:27.749921 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.749879    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"]
Nov 03 16:33:27.836699 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.836663    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:27.860085 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.860044    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:27.860272 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.860177    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wqxv\" (UniqueName: \"kubernetes.io/projected/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-kube-api-access-8wqxv\") pod \"maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:27.860272 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.860225    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:27.860388 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.860350    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:27.961125 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.961070    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:27.961373 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.961191    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:27.961373 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.961273    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:27.961373 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.961332    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wqxv\" (UniqueName: \"kubernetes.io/projected/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-kube-api-access-8wqxv\") pod \"maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:27.961896 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.961867    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:27.964347 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.964300    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:27.964976 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.964954    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:27.975057 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:27.975007    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wqxv\" (UniqueName: \"kubernetes.io/projected/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-kube-api-access-8wqxv\") pod \"maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:28.038323 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.038291    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:28.085831 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.082587    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:33:28.085831 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.084609    6506 generic.go:356] "Generic (PLEG): container finished" podID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerID="bc4429adad9428ed1eb2bda8566b67a254ccbd7aca5a1350e1ba74c096679a3c" exitCode=0
Nov 03 16:33:28.085831 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.084676    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" event={"ID":"9df80d95-49e6-476a-a626-0dd06e6afb66","Type":"ContainerDied","Data":"bc4429adad9428ed1eb2bda8566b67a254ccbd7aca5a1350e1ba74c096679a3c"}
Nov 03 16:33:28.086251 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.086231    6506 generic.go:356] "Generic (PLEG): container finished" podID="13e172cb-f664-4da0-a922-08b5a0eb29d6" containerID="2ca18f7e3ca5b61c9d444d0e44c38cbade52bd818c7123d2e0fec5f33688cac6" exitCode=0
Nov 03 16:33:28.086360 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.086298    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"
Nov 03 16:33:28.086360 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.086329    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" event={"ID":"13e172cb-f664-4da0-a922-08b5a0eb29d6","Type":"ContainerDied","Data":"2ca18f7e3ca5b61c9d444d0e44c38cbade52bd818c7123d2e0fec5f33688cac6"}
Nov 03 16:33:28.086462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.086380    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb" event={"ID":"13e172cb-f664-4da0-a922-08b5a0eb29d6","Type":"ContainerDied","Data":"65972e20bbe61f794441111bafae8ed3225e3c436fe9890cedd2c764b73287e3"}
Nov 03 16:33:28.086462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.086410    6506 scope.go:122] "RemoveContainer" containerID="2ca18f7e3ca5b61c9d444d0e44c38cbade52bd818c7123d2e0fec5f33688cac6"
Nov 03 16:33:28.100446 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.099488    6506 scope.go:122] "RemoveContainer" containerID="2ca18f7e3ca5b61c9d444d0e44c38cbade52bd818c7123d2e0fec5f33688cac6"
Nov 03 16:33:28.100446 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.100379    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:28.102666 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:33:28.101523    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca18f7e3ca5b61c9d444d0e44c38cbade52bd818c7123d2e0fec5f33688cac6\": container with ID starting with 2ca18f7e3ca5b61c9d444d0e44c38cbade52bd818c7123d2e0fec5f33688cac6 not found: ID does not exist" containerID="2ca18f7e3ca5b61c9d444d0e44c38cbade52bd818c7123d2e0fec5f33688cac6"
Nov 03 16:33:28.102666 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.101561    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca18f7e3ca5b61c9d444d0e44c38cbade52bd818c7123d2e0fec5f33688cac6"} err="failed to get container status \"2ca18f7e3ca5b61c9d444d0e44c38cbade52bd818c7123d2e0fec5f33688cac6\": rpc error: code = NotFound desc = could not find container \"2ca18f7e3ca5b61c9d444d0e44c38cbade52bd818c7123d2e0fec5f33688cac6\": container with ID starting with 2ca18f7e3ca5b61c9d444d0e44c38cbade52bd818c7123d2e0fec5f33688cac6 not found: ID does not exist"
Nov 03 16:33:28.135348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.135319    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:33:28.264759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.264676    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-workload-socket\") pod \"9df80d95-49e6-476a-a626-0dd06e6afb66\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") "
Nov 03 16:33:28.264759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.264712    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-credential-socket\") pod \"13e172cb-f664-4da0-a922-08b5a0eb29d6\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") "
Nov 03 16:33:28.264759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.264734    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-podinfo\") pod \"9df80d95-49e6-476a-a626-0dd06e6afb66\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") "
Nov 03 16:33:28.265068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.264767    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-workload-certs\") pod \"13e172cb-f664-4da0-a922-08b5a0eb29d6\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") "
Nov 03 16:33:28.265068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.264848    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-credential-socket\") pod \"9df80d95-49e6-476a-a626-0dd06e6afb66\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") "
Nov 03 16:33:28.265068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.264880    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df80d95-49e6-476a-a626-0dd06e6afb66-kube-api-access-p8pn9\" (UniqueName: \"kubernetes.io/projected/9df80d95-49e6-476a-a626-0dd06e6afb66-kube-api-access-p8pn9\") pod \"9df80d95-49e6-476a-a626-0dd06e6afb66\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") "
Nov 03 16:33:28.265068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.264911    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-envoy\") pod \"13e172cb-f664-4da0-a922-08b5a0eb29d6\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") "
Nov 03 16:33:28.265068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.264938    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-token\" (UniqueName: \"kubernetes.io/projected/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-token\") pod \"13e172cb-f664-4da0-a922-08b5a0eb29d6\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") "
Nov 03 16:33:28.265068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.264970    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-podinfo\") pod \"13e172cb-f664-4da0-a922-08b5a0eb29d6\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") "
Nov 03 16:33:28.265068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265000    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-token\" (UniqueName: \"kubernetes.io/projected/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-token\") pod \"9df80d95-49e6-476a-a626-0dd06e6afb66\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") "
Nov 03 16:33:28.265068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265040    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-data\") pod \"9df80d95-49e6-476a-a626-0dd06e6afb66\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") "
Nov 03 16:33:28.265068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265015    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-credential-socket" pod "13e172cb-f664-4da0-a922-08b5a0eb29d6" (UID: "13e172cb-f664-4da0-a922-08b5a0eb29d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:28.265068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265008    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-workload-socket" pod "9df80d95-49e6-476a-a626-0dd06e6afb66" (UID: "9df80d95-49e6-476a-a626-0dd06e6afb66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:28.265068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265070    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-envoy\") pod \"9df80d95-49e6-476a-a626-0dd06e6afb66\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") "
Nov 03 16:33:28.265725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265085    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-workload-certs" pod "13e172cb-f664-4da0-a922-08b5a0eb29d6" (UID: "13e172cb-f664-4da0-a922-08b5a0eb29d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:28.265725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265117    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-data\") pod \"13e172cb-f664-4da0-a922-08b5a0eb29d6\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") "
Nov 03 16:33:28.265725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265166    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e172cb-f664-4da0-a922-08b5a0eb29d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e172cb-f664-4da0-a922-08b5a0eb29d6-istiod-ca-cert\") pod \"13e172cb-f664-4da0-a922-08b5a0eb29d6\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") "
Nov 03 16:33:28.265725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265254    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df80d95-49e6-476a-a626-0dd06e6afb66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df80d95-49e6-476a-a626-0dd06e6afb66-istiod-ca-cert\") pod \"9df80d95-49e6-476a-a626-0dd06e6afb66\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") "
Nov 03 16:33:28.265725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265293    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-workload-certs\") pod \"9df80d95-49e6-476a-a626-0dd06e6afb66\" (UID: \"9df80d95-49e6-476a-a626-0dd06e6afb66\") "
Nov 03 16:33:28.265725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265331    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-workload-socket\") pod \"13e172cb-f664-4da0-a922-08b5a0eb29d6\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") "
Nov 03 16:33:28.265725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265366    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e172cb-f664-4da0-a922-08b5a0eb29d6-kube-api-access-sqglf\" (UniqueName: \"kubernetes.io/projected/13e172cb-f664-4da0-a922-08b5a0eb29d6-kube-api-access-sqglf\") pod \"13e172cb-f664-4da0-a922-08b5a0eb29d6\" (UID: \"13e172cb-f664-4da0-a922-08b5a0eb29d6\") "
Nov 03 16:33:28.265725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265438    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-credential-socket" pod "9df80d95-49e6-476a-a626-0dd06e6afb66" (UID: "9df80d95-49e6-476a-a626-0dd06e6afb66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:28.266356 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265895    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.266356 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265918    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.266356 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265923    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e172cb-f664-4da0-a922-08b5a0eb29d6-istiod-ca-cert" pod "13e172cb-f664-4da0-a922-08b5a0eb29d6" (UID: "13e172cb-f664-4da0-a922-08b5a0eb29d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:33:28.266356 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265935    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.266356 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.265951    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.266356 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.266154    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-data" pod "9df80d95-49e6-476a-a626-0dd06e6afb66" (UID: "9df80d95-49e6-476a-a626-0dd06e6afb66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:28.266662 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.266390    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df80d95-49e6-476a-a626-0dd06e6afb66-istiod-ca-cert" pod "9df80d95-49e6-476a-a626-0dd06e6afb66" (UID: "9df80d95-49e6-476a-a626-0dd06e6afb66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:33:28.266723 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.266675    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-workload-certs" pod "9df80d95-49e6-476a-a626-0dd06e6afb66" (UID: "9df80d95-49e6-476a-a626-0dd06e6afb66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:28.266778 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.266749    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-workload-socket" pod "13e172cb-f664-4da0-a922-08b5a0eb29d6" (UID: "13e172cb-f664-4da0-a922-08b5a0eb29d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:28.266854 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.266817    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-data" pod "13e172cb-f664-4da0-a922-08b5a0eb29d6" (UID: "13e172cb-f664-4da0-a922-08b5a0eb29d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:28.268304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.268274    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-token" pod "9df80d95-49e6-476a-a626-0dd06e6afb66" (UID: "9df80d95-49e6-476a-a626-0dd06e6afb66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:33:28.268593 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.268568    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-token" pod "13e172cb-f664-4da0-a922-08b5a0eb29d6" (UID: "13e172cb-f664-4da0-a922-08b5a0eb29d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:33:28.268593 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.268577    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df80d95-49e6-476a-a626-0dd06e6afb66-kube-api-access-p8pn9" pod "9df80d95-49e6-476a-a626-0dd06e6afb66" (UID: "9df80d95-49e6-476a-a626-0dd06e6afb66"). InnerVolumeSpecName "kube-api-access-p8pn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:33:28.268728 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.268678    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-podinfo" pod "9df80d95-49e6-476a-a626-0dd06e6afb66" (UID: "9df80d95-49e6-476a-a626-0dd06e6afb66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:33:28.268784 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.268760    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-podinfo" pod "13e172cb-f664-4da0-a922-08b5a0eb29d6" (UID: "13e172cb-f664-4da0-a922-08b5a0eb29d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:33:28.269207 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.269181    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-envoy" pod "13e172cb-f664-4da0-a922-08b5a0eb29d6" (UID: "13e172cb-f664-4da0-a922-08b5a0eb29d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:28.269385 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.269367    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e172cb-f664-4da0-a922-08b5a0eb29d6-kube-api-access-sqglf" pod "13e172cb-f664-4da0-a922-08b5a0eb29d6" (UID: "13e172cb-f664-4da0-a922-08b5a0eb29d6"). InnerVolumeSpecName "kube-api-access-sqglf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:33:28.269843 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.269816    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-envoy" pod "9df80d95-49e6-476a-a626-0dd06e6afb66" (UID: "9df80d95-49e6-476a-a626-0dd06e6afb66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:28.272647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.272625    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:28.344751 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.344725    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"]
Nov 03 16:33:28.346758 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:33:28.346729    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681f7fb0_70a4_4865_83e3_ad0c0d07cc48.slice/crio-1c76c5b468e1fc7425871af1c845a29c59882ed5a0b13645e71c1e62fb5abfda WatchSource:0}: Error finding container 1c76c5b468e1fc7425871af1c845a29c59882ed5a0b13645e71c1e62fb5abfda: Status 404 returned error can't find the container with id 1c76c5b468e1fc7425871af1c845a29c59882ed5a0b13645e71c1e62fb5abfda
Nov 03 16:33:28.348058 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.348041    6506 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:33:28.366880 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.366854    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.366880 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.366878    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8pn9\" (UniqueName: \"kubernetes.io/projected/9df80d95-49e6-476a-a626-0dd06e6afb66-kube-api-access-p8pn9\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.366994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.366890    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.366994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.366899    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.366994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.366906    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.366994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.366914    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.366994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.366922    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.366994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.366931    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.366994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.366938    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.366994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.366946    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e172cb-f664-4da0-a922-08b5a0eb29d6-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.366994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.366956    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df80d95-49e6-476a-a626-0dd06e6afb66-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.366994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.366963    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df80d95-49e6-476a-a626-0dd06e6afb66-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.366994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.366972    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e172cb-f664-4da0-a922-08b5a0eb29d6-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.366994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.366981    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqglf\" (UniqueName: \"kubernetes.io/projected/13e172cb-f664-4da0-a922-08b5a0eb29d6-kube-api-access-sqglf\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.411575 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.411550    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"]
Nov 03 16:33:28.413576 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.413550    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qqwsb"]
Nov 03 16:33:28.795845 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:33:28.795782    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:33:28.842863 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:28.842835    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:29.092775 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:29.092698    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"
Nov 03 16:33:29.092775 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:29.092744    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg" event={"ID":"9df80d95-49e6-476a-a626-0dd06e6afb66","Type":"ContainerDied","Data":"d7f875355354af0bbd923b6959f1467b29da5b66ff910f767243ab497d0edfa5"}
Nov 03 16:33:29.093034 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:29.092839    6506 scope.go:122] "RemoveContainer" containerID="bc4429adad9428ed1eb2bda8566b67a254ccbd7aca5a1350e1ba74c096679a3c"
Nov 03 16:33:29.095693 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:29.095667    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf" event={"ID":"681f7fb0-70a4-4865-83e3-ad0c0d07cc48","Type":"ContainerStarted","Data":"84875c23008d9def1ef46dd3cce228505e582632c22a98e8c6c1dcd92b116789"}
Nov 03 16:33:29.095829 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:29.095714    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf" event={"ID":"681f7fb0-70a4-4865-83e3-ad0c0d07cc48","Type":"ContainerStarted","Data":"1c76c5b468e1fc7425871af1c845a29c59882ed5a0b13645e71c1e62fb5abfda"}
Nov 03 16:33:29.095829 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:29.095806    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:29.132950 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:29.132911    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf" podStartSLOduration=1.676674736 podStartE2EDuration="2.13289796s" totalImagesPullingTime="456.223224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:29.131765622 +0000 UTC m=+1823.145163770" watchObservedRunningTime="2026-11-03 16:33:29.13289796 +0000 UTC m=+1823.146296105"
Nov 03 16:33:29.150680 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:29.150655    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"]
Nov 03 16:33:29.152134 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:29.152116    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-n9mvg"]
Nov 03 16:33:29.805291 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:29.805259    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:30.108562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:30.108484    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e172cb-f664-4da0-a922-08b5a0eb29d6" path="/var/lib/kubelet/pods/13e172cb-f664-4da0-a922-08b5a0eb29d6/volumes"
Nov 03 16:33:30.108991 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:30.108975    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" path="/var/lib/kubelet/pods/9df80d95-49e6-476a-a626-0dd06e6afb66/volumes"
Nov 03 16:33:30.539378 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:30.539337    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:31.549330 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:31.549299    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:32.417580 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:32.417544    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:33.248196 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:33.248165    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:34.026357 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:34.026328    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:34.822369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:34.822333    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:35.114312 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:35.114247    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:33:36.143499 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:36.143467    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:36.344638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:36.344606    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:36.903647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:36.903609    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:37.075592 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.075560    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:37.242965 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.242893    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:37.887191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.887161    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"]
Nov 03 16:33:37.887505 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.887492    6506 cpu_manager.go:414] "containerMap: removing container" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerName="istio-proxy"
Nov 03 16:33:37.887554 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.887507    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerName="istio-proxy"
Nov 03 16:33:37.887554 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.887530    6506 cpu_manager.go:414] "containerMap: removing container" podUID="13e172cb-f664-4da0-a922-08b5a0eb29d6" containerName="istio-proxy"
Nov 03 16:33:37.887554 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.887536    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e172cb-f664-4da0-a922-08b5a0eb29d6" containerName="istio-proxy"
Nov 03 16:33:37.887654 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.887607    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df80d95-49e6-476a-a626-0dd06e6afb66" containerName="istio-proxy"
Nov 03 16:33:37.887654 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.887620    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e172cb-f664-4da0-a922-08b5a0eb29d6" containerName="istio-proxy"
Nov 03 16:33:37.892254 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.892232    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:37.901069 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.901052    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-s22x6\""
Nov 03 16:33:37.916614 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.916592    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 16:33:37.917193 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.917172    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"]
Nov 03 16:33:37.945404 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.945367    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:37.945404 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.945406    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:37.945635 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.945431    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:37.945635 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.945518    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shvqh\" (UniqueName: \"kubernetes.io/projected/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-kube-api-access-shvqh\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:37.945635 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.945574    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:37.945635 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.945604    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:37.945908 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.945647    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:37.945908 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.945681    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:37.945908 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.945703    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:37.978900 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:37.978861    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:38.046452 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.046420    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.046452 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.046454    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.046675 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.046478    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.046675 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.046498    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shvqh\" (UniqueName: \"kubernetes.io/projected/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-kube-api-access-shvqh\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.046675 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.046614    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.046675 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.046648    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.046939 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.046692    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.046939 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.046719    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.046939 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.046758    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.047077 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.046947    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.047077 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.046988    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.047148 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.047128    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.047199 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.047182    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.047765 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.047743    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.049092 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.049066    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.049199 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.049175    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.059989 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.059969    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shvqh\" (UniqueName: \"kubernetes.io/projected/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-kube-api-access-shvqh\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.060080 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.060018    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-pq785\" (UID: \"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.205317 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.205239    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:38.492732 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.492708    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"]
Nov 03 16:33:38.494873 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:33:38.494847    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc4ed21_1d34_4dd0_9848_e3ebf48fce9f.slice/crio-e927d9882d96a674e3f88a4582c27d98e052d803036369c350a09e601828db4f WatchSource:0}: Error finding container e927d9882d96a674e3f88a4582c27d98e052d803036369c350a09e601828db4f: Status 404 returned error can't find the container with id e927d9882d96a674e3f88a4582c27d98e052d803036369c350a09e601828db4f
Nov 03 16:33:38.501085 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.501049    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:33:38.501218 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.501191    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:33:38.501308 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.501289    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:33:38.773424 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:38.773397    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:39.154768 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.154667    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785" event={"ID":"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f","Type":"ContainerStarted","Data":"f943938432833e0162d7ba61a040b4b56c83541ea3364574656a54838918b4b5"}
Nov 03 16:33:39.154768 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.154762    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785" event={"ID":"9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f","Type":"ContainerStarted","Data":"e927d9882d96a674e3f88a4582c27d98e052d803036369c350a09e601828db4f"}
Nov 03 16:33:39.180159 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.180101    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785" podStartSLOduration=2.180083721 podStartE2EDuration="2.180083721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:39.17952504 +0000 UTC m=+1833.192923189" watchObservedRunningTime="2026-11-03 16:33:39.180083721 +0000 UTC m=+1833.193481869"
Nov 03 16:33:39.205661 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.205634    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:39.208308 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.208283    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-pq785 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 16:33:39.211815 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.208489    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785" podUID="9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 16:33:39.660304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.660273    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"]
Nov 03 16:33:39.664138 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.664117    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.670841 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.670782    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ba8f54\"/\"kube-root-ca.crt\""
Nov 03 16:33:39.672098 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.672076    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ba8f54\"/\"openshift-service-ca.crt\""
Nov 03 16:33:39.672379 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.672362    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ba8f54\"/\"default-dockercfg-57vvd\""
Nov 03 16:33:39.672567 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.672538    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ba8f54\"/\"simulator-single-main-ba8f54-kserve-self-signed-certs\""
Nov 03 16:33:39.680090 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.680066    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"]
Nov 03 16:33:39.759992 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.759952    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtl9q\" (UniqueName: \"kubernetes.io/projected/2e84614a-8475-431c-a256-df6c8006deaf-kube-api-access-jtl9q\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.760186 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.760006    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-model-cache\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.760186 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.760039    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e84614a-8475-431c-a256-df6c8006deaf-tls-certs\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.760186 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.760171    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-dshm\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.760372 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.760198    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-tmp-dir\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.760372 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.760256    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-home\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.861557 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.861495    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-model-cache\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.861749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.861575    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e84614a-8475-431c-a256-df6c8006deaf-tls-certs\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.861749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.861601    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-dshm\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.861749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.861616    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-tmp-dir\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.861749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.861638    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-home\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.862014 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.861750    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtl9q\" (UniqueName: \"kubernetes.io/projected/2e84614a-8475-431c-a256-df6c8006deaf-kube-api-access-jtl9q\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.862014 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.861981    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-model-cache\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.862118 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.862060    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-tmp-dir\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.862118 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.862074    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-home\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.864068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.864040    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-dshm\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.864281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.864263    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e84614a-8475-431c-a256-df6c8006deaf-tls-certs\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.875446 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.875423    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtl9q\" (UniqueName: \"kubernetes.io/projected/2e84614a-8475-431c-a256-df6c8006deaf-kube-api-access-jtl9q\") pod \"simulator-single-main-ba8f54-kserve-7477d59b7b-596xv\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") " pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:39.977987 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:39.977905    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:40.206565 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:40.206531    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-pq785 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 16:33:40.206782 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:40.206602    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785" podUID="9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 16:33:40.290129 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:40.290104    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"]
Nov 03 16:33:40.292165 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:33:40.292140    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e84614a_8475_431c_a256_df6c8006deaf.slice/crio-b212307e9e6949cf00a79211138173df76fbc3fc47a014cbe9569ffff2d30c10 WatchSource:0}: Error finding container b212307e9e6949cf00a79211138173df76fbc3fc47a014cbe9569ffff2d30c10: Status 404 returned error can't find the container with id b212307e9e6949cf00a79211138173df76fbc3fc47a014cbe9569ffff2d30c10
Nov 03 16:33:40.790752 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:40.790719    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:40.975027 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:40.974992    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:41.166478 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:41.166383    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv" event={"ID":"2e84614a-8475-431c-a256-df6c8006deaf","Type":"ContainerStarted","Data":"c08334efbf800b22adeda05a4abdc31c55173a07044b39d763b4fbf75ba357fa"}
Nov 03 16:33:41.166478 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:41.166431    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv" event={"ID":"2e84614a-8475-431c-a256-df6c8006deaf","Type":"ContainerStarted","Data":"b212307e9e6949cf00a79211138173df76fbc3fc47a014cbe9569ffff2d30c10"}
Nov 03 16:33:41.188954 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:41.188888    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv" podStartSLOduration=2.188873953 podStartE2EDuration="2.188873953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:41.188158763 +0000 UTC m=+1835.201556910" watchObservedRunningTime="2026-11-03 16:33:41.188873953 +0000 UTC m=+1835.202272096"
Nov 03 16:33:41.206529 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:41.206497    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-pq785 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 16:33:41.206674 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:41.206550    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785" podUID="9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 16:33:41.780372 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:41.780337    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:41.978553 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:41.978515    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:42.206594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:42.206517    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-pq785 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 16:33:42.206594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:42.206581    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785" podUID="9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 16:33:42.884919 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:42.884885    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:43.205621 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:43.205543    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-pq785 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 16:33:43.205621 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:43.205592    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785" podUID="9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 16:33:43.886388 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:43.886351    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:44.206312 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:44.206232    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-pq785 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 16:33:44.206312 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:44.206290    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785" podUID="9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 16:33:44.505837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:44.505723    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:44.829271 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:44.829230    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:45.206205 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:45.206117    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-pq785 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 16:33:45.206205 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:45.206175    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785" podUID="9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 16:33:45.539342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:45.539307    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:46.519440 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:46.519408    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:46.769587 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:46.769466    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:46.769857 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:46.769839    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:46.770731 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:46.770708    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785"
Nov 03 16:33:47.353468 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:47.353434    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:48.242932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:48.242902    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:48.929416 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:48.929385    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:49.734754 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:49.734718    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:49.978710 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:49.978670    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:49.991299 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:49.991230    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:50.229436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:50.229410    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:33:50.416390 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:50.416353    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:51.128296 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:51.128229    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:51.808759 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:51.808722    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:52.534877 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:52.534840    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:53.217407 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:53.217370    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:54.021599 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:54.021565    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:54.923655 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:54.923614    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:55.613631 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:55.613597    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:56.428776 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:56.428738    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:57.232783 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:57.232752    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:58.102672 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:58.102634    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:58.797206 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:58.797175    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:33:58.797646 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:33:58.797620    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:33:59.509599 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:33:59.509571    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:00.201784 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:00.201752    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:00.911001 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:00.910967    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:01.622104 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:01.622072    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:02.415062 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:02.415018    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:03.131910 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:03.131871    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:03.826464 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:03.826431    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:04.696646 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:04.696609    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:05.399942 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:05.399907    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:06.123568 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:06.123539    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:06.859158 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:06.859126    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:06.975076 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:06.975044    6506 scope.go:122] "RemoveContainer" containerID="35fd670030fe8c37aeb66bb9db7595b4df45581494c0c8eb9f198af832c989d8"
Nov 03 16:34:06.983627 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:06.983609    6506 scope.go:122] "RemoveContainer" containerID="38641aff7c507c09270d61fe047cd54a800e8d570f338d7f8ab7fb2a2ecab382"
Nov 03 16:34:07.641472 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:07.641438    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:08.520421 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:08.520385    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:09.291743 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:09.291707    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:10.007699 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:10.007638    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:10.432845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:10.432815    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:10.788626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:10.788586    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:11.602419 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:11.602375    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:12.309344 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:12.309308    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:13.026454 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:13.026425    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:13.722401 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:13.722367    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:14.415656 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:14.415623    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:15.130881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:15.130845    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:15.871990 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:15.871954    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:16.520198 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:16.520151    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:17.302336 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:17.302272    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:18.006313 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:18.006271    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:18.702068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:18.702031    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:19.512046 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:19.511991    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:19.951223 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:19.951190    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:20.373626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:20.373590    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:21.391716 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:21.391672    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:22.255930 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:22.255892    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:22.996376 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:22.996339    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:23.713834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:23.713768    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:24.464900 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:24.464863    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:24.851453 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:24.851422    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"]
Nov 03 16:34:24.978423 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:24.978390    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:25.177361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:25.177285    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:25.809029 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:25.808997    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:26.611304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:26.611272    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:27.307215 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:27.307182    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:28.007820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:28.007776    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:28.706450 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:28.706412    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:28.795277 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:34:28.795238    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:34:29.400266 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:29.400221    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:30.103857 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:30.103827    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:30.382399 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:30.382320    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:30.541808 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:30.541754    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:30.807786 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:30.807748    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:31.304260 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.304224    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"]
Nov 03 16:34:31.305437 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.305394    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf" podUID="681f7fb0-70a4-4865-83e3-ad0c0d07cc48" containerName="maas-api" containerID="cri-o://84875c23008d9def1ef46dd3cce228505e582632c22a98e8c6c1dcd92b116789" gracePeriod=30
Nov 03 16:34:31.401726 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.401692    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:31.541395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.541364    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:34:31.569352 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.569229    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:31.640210 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.640183    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-metrics-certs\") pod \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") "
Nov 03 16:34:31.640378 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.640237    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-openshift-service-ca\") pod \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") "
Nov 03 16:34:31.640378 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.640297    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-maas-api-tls\") pod \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") "
Nov 03 16:34:31.640378 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.640331    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-kube-api-access-8wqxv\" (UniqueName: \"kubernetes.io/projected/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-kube-api-access-8wqxv\") pod \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\" (UID: \"681f7fb0-70a4-4865-83e3-ad0c0d07cc48\") "
Nov 03 16:34:31.640641 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.640620    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-openshift-service-ca" pod "681f7fb0-70a4-4865-83e3-ad0c0d07cc48" (UID: "681f7fb0-70a4-4865-83e3-ad0c0d07cc48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:34:31.642458 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.642422    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-metrics-certs" pod "681f7fb0-70a4-4865-83e3-ad0c0d07cc48" (UID: "681f7fb0-70a4-4865-83e3-ad0c0d07cc48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:34:31.642567 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.642542    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-kube-api-access-8wqxv" pod "681f7fb0-70a4-4865-83e3-ad0c0d07cc48" (UID: "681f7fb0-70a4-4865-83e3-ad0c0d07cc48"). InnerVolumeSpecName "kube-api-access-8wqxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:34:31.642567 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.642554    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-maas-api-tls" pod "681f7fb0-70a4-4865-83e3-ad0c0d07cc48" (UID: "681f7fb0-70a4-4865-83e3-ad0c0d07cc48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:34:31.725565 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.725522    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:31.741364 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.741342    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-maas-api-tls\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:31.741451 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.741363    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wqxv\" (UniqueName: \"kubernetes.io/projected/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-kube-api-access-8wqxv\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:31.741451 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.741375    6506 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-metrics-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:31.741451 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:31.741383    6506 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/681f7fb0-70a4-4865-83e3-ad0c0d07cc48-openshift-service-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:32.205193 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:32.205157    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:32.420841 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:32.420715    6506 generic.go:356] "Generic (PLEG): container finished" podID="681f7fb0-70a4-4865-83e3-ad0c0d07cc48" containerID="84875c23008d9def1ef46dd3cce228505e582632c22a98e8c6c1dcd92b116789" exitCode=0
Nov 03 16:34:32.420841 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:32.420772    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"
Nov 03 16:34:32.421374 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:32.420831    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf" event={"ID":"681f7fb0-70a4-4865-83e3-ad0c0d07cc48","Type":"ContainerDied","Data":"84875c23008d9def1ef46dd3cce228505e582632c22a98e8c6c1dcd92b116789"}
Nov 03 16:34:32.421374 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:32.420900    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf" event={"ID":"681f7fb0-70a4-4865-83e3-ad0c0d07cc48","Type":"ContainerDied","Data":"1c76c5b468e1fc7425871af1c845a29c59882ed5a0b13645e71c1e62fb5abfda"}
Nov 03 16:34:32.421374 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:32.420923    6506 scope.go:122] "RemoveContainer" containerID="84875c23008d9def1ef46dd3cce228505e582632c22a98e8c6c1dcd92b116789"
Nov 03 16:34:32.430374 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:32.430356    6506 scope.go:122] "RemoveContainer" containerID="84875c23008d9def1ef46dd3cce228505e582632c22a98e8c6c1dcd92b116789"
Nov 03 16:34:32.430643 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:34:32.430626    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84875c23008d9def1ef46dd3cce228505e582632c22a98e8c6c1dcd92b116789\": container with ID starting with 84875c23008d9def1ef46dd3cce228505e582632c22a98e8c6c1dcd92b116789 not found: ID does not exist" containerID="84875c23008d9def1ef46dd3cce228505e582632c22a98e8c6c1dcd92b116789"
Nov 03 16:34:32.430697 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:32.430653    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84875c23008d9def1ef46dd3cce228505e582632c22a98e8c6c1dcd92b116789"} err="failed to get container status \"84875c23008d9def1ef46dd3cce228505e582632c22a98e8c6c1dcd92b116789\": rpc error: code = NotFound desc = could not find container \"84875c23008d9def1ef46dd3cce228505e582632c22a98e8c6c1dcd92b116789\": container with ID starting with 84875c23008d9def1ef46dd3cce228505e582632c22a98e8c6c1dcd92b116789 not found: ID does not exist"
Nov 03 16:34:32.441362 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:32.441339    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"]
Nov 03 16:34:32.445967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:32.445946    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ba8f54-cb4b7fc96-lbpsf"]
Nov 03 16:34:34.102356 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:34.102317    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681f7fb0-70a4-4865-83e3-ad0c0d07cc48" path="/var/lib/kubelet/pods/681f7fb0-70a4-4865-83e3-ad0c0d07cc48/volumes"
Nov 03 16:34:38.280655 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:38.280618    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:38.445474 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:38.445438    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:38.643113 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:38.643033    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:34:39.838558 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.838524    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"]
Nov 03 16:34:39.846688 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.846660    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf5xw"]
Nov 03 16:34:39.869442 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.869401    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"]
Nov 03 16:34:39.869810 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.869780    6506 cpu_manager.go:414] "containerMap: removing container" podUID="681f7fb0-70a4-4865-83e3-ad0c0d07cc48" containerName="maas-api"
Nov 03 16:34:39.869878 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.869809    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681f7fb0-70a4-4865-83e3-ad0c0d07cc48" containerName="maas-api"
Nov 03 16:34:39.869925 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.869903    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="681f7fb0-70a4-4865-83e3-ad0c0d07cc48" containerName="maas-api"
Nov 03 16:34:39.871470 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.871441    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv" podUID="2e84614a-8475-431c-a256-df6c8006deaf" containerName="main" containerID="cri-o://c08334efbf800b22adeda05a4abdc31c55173a07044b39d763b4fbf75ba357fa" gracePeriod=45
Nov 03 16:34:39.873858 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.873842    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"
Nov 03 16:34:39.880908 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.880561    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:34:39.881619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.881598    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:34:39.881717 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.881681    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:34:39.881717 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.881699    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:34:39.883994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.883971    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"]
Nov 03 16:34:39.915526 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.915503    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136e618c-25fb-4f4d-bdcd-13af4f832c34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nt4jl\" (UID: \"136e618c-25fb-4f4d-bdcd-13af4f832c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"
Nov 03 16:34:39.915616 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.915569    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfxf9\" (UniqueName: \"kubernetes.io/projected/136e618c-25fb-4f4d-bdcd-13af4f832c34-kube-api-access-pfxf9\") pod \"maas-api-revoke-sub-3d4156f1-nt4jl\" (UID: \"136e618c-25fb-4f4d-bdcd-13af4f832c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"
Nov 03 16:34:39.915672 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:39.915654    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136e618c-25fb-4f4d-bdcd-13af4f832c34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nt4jl\" (UID: \"136e618c-25fb-4f4d-bdcd-13af4f832c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"
Nov 03 16:34:40.016895 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.016849    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136e618c-25fb-4f4d-bdcd-13af4f832c34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nt4jl\" (UID: \"136e618c-25fb-4f4d-bdcd-13af4f832c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"
Nov 03 16:34:40.017062 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.016965    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfxf9\" (UniqueName: \"kubernetes.io/projected/136e618c-25fb-4f4d-bdcd-13af4f832c34-kube-api-access-pfxf9\") pod \"maas-api-revoke-sub-3d4156f1-nt4jl\" (UID: \"136e618c-25fb-4f4d-bdcd-13af4f832c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"
Nov 03 16:34:40.017062 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.017009    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136e618c-25fb-4f4d-bdcd-13af4f832c34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nt4jl\" (UID: \"136e618c-25fb-4f4d-bdcd-13af4f832c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"
Nov 03 16:34:40.017967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.017935    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136e618c-25fb-4f4d-bdcd-13af4f832c34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nt4jl\" (UID: \"136e618c-25fb-4f4d-bdcd-13af4f832c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"
Nov 03 16:34:40.019882 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.019862    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136e618c-25fb-4f4d-bdcd-13af4f832c34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nt4jl\" (UID: \"136e618c-25fb-4f4d-bdcd-13af4f832c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"
Nov 03 16:34:40.030938 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.030910    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfxf9\" (UniqueName: \"kubernetes.io/projected/136e618c-25fb-4f4d-bdcd-13af4f832c34-kube-api-access-pfxf9\") pod \"maas-api-revoke-sub-3d4156f1-nt4jl\" (UID: \"136e618c-25fb-4f4d-bdcd-13af4f832c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"
Nov 03 16:34:40.103310 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.103282    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5144d016-0056-4b27-b448-6eadf8287a7a" path="/var/lib/kubelet/pods/5144d016-0056-4b27-b448-6eadf8287a7a/volumes"
Nov 03 16:34:40.123262 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.123238    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:34:40.218379 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.218336    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e84614a-8475-431c-a256-df6c8006deaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e84614a-8475-431c-a256-df6c8006deaf-tls-certs\") pod \"2e84614a-8475-431c-a256-df6c8006deaf\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") "
Nov 03 16:34:40.218582 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.218426    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-home\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-home\") pod \"2e84614a-8475-431c-a256-df6c8006deaf\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") "
Nov 03 16:34:40.218663 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.218611    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-tmp-dir\") pod \"2e84614a-8475-431c-a256-df6c8006deaf\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") "
Nov 03 16:34:40.218716 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.218656    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-home" pod "2e84614a-8475-431c-a256-df6c8006deaf" (UID: "2e84614a-8475-431c-a256-df6c8006deaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:34:40.218716 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.218693    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e84614a-8475-431c-a256-df6c8006deaf-kube-api-access-jtl9q\" (UniqueName: \"kubernetes.io/projected/2e84614a-8475-431c-a256-df6c8006deaf-kube-api-access-jtl9q\") pod \"2e84614a-8475-431c-a256-df6c8006deaf\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") "
Nov 03 16:34:40.218846 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.218740    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-dshm\") pod \"2e84614a-8475-431c-a256-df6c8006deaf\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") "
Nov 03 16:34:40.218846 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.218783    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-model-cache\") pod \"2e84614a-8475-431c-a256-df6c8006deaf\" (UID: \"2e84614a-8475-431c-a256-df6c8006deaf\") "
Nov 03 16:34:40.218974 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.218850    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-tmp-dir" pod "2e84614a-8475-431c-a256-df6c8006deaf" (UID: "2e84614a-8475-431c-a256-df6c8006deaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:34:40.220118 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.219125    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-model-cache" pod "2e84614a-8475-431c-a256-df6c8006deaf" (UID: "2e84614a-8475-431c-a256-df6c8006deaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:34:40.220118 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.219154    6506 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-home\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:40.220118 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.219174    6506 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-tmp-dir\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:40.220934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.220909    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-dshm" pod "2e84614a-8475-431c-a256-df6c8006deaf" (UID: "2e84614a-8475-431c-a256-df6c8006deaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:34:40.220934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.220918    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e84614a-8475-431c-a256-df6c8006deaf-tls-certs" pod "2e84614a-8475-431c-a256-df6c8006deaf" (UID: "2e84614a-8475-431c-a256-df6c8006deaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:34:40.221196 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.221180    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e84614a-8475-431c-a256-df6c8006deaf-kube-api-access-jtl9q" pod "2e84614a-8475-431c-a256-df6c8006deaf" (UID: "2e84614a-8475-431c-a256-df6c8006deaf"). InnerVolumeSpecName "kube-api-access-jtl9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:34:40.247914 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.247889    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"
Nov 03 16:34:40.320456 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.320426    6506 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-dshm\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:40.320456 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.320456    6506 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e84614a-8475-431c-a256-df6c8006deaf-model-cache\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:40.320673 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.320466    6506 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e84614a-8475-431c-a256-df6c8006deaf-tls-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:40.320673 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.320477    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtl9q\" (UniqueName: \"kubernetes.io/projected/2e84614a-8475-431c-a256-df6c8006deaf-kube-api-access-jtl9q\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:40.461629 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.461541    6506 generic.go:356] "Generic (PLEG): container finished" podID="2e84614a-8475-431c-a256-df6c8006deaf" containerID="c08334efbf800b22adeda05a4abdc31c55173a07044b39d763b4fbf75ba357fa" exitCode=0
Nov 03 16:34:40.461629 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.461612    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"
Nov 03 16:34:40.461886 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.461626    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv" event={"ID":"2e84614a-8475-431c-a256-df6c8006deaf","Type":"ContainerDied","Data":"c08334efbf800b22adeda05a4abdc31c55173a07044b39d763b4fbf75ba357fa"}
Nov 03 16:34:40.461886 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.461668    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv" event={"ID":"2e84614a-8475-431c-a256-df6c8006deaf","Type":"ContainerDied","Data":"b212307e9e6949cf00a79211138173df76fbc3fc47a014cbe9569ffff2d30c10"}
Nov 03 16:34:40.461886 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.461687    6506 scope.go:122] "RemoveContainer" containerID="c08334efbf800b22adeda05a4abdc31c55173a07044b39d763b4fbf75ba357fa"
Nov 03 16:34:40.470727 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.470701    6506 scope.go:122] "RemoveContainer" containerID="c08334efbf800b22adeda05a4abdc31c55173a07044b39d763b4fbf75ba357fa"
Nov 03 16:34:40.471037 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:34:40.471018    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08334efbf800b22adeda05a4abdc31c55173a07044b39d763b4fbf75ba357fa\": container with ID starting with c08334efbf800b22adeda05a4abdc31c55173a07044b39d763b4fbf75ba357fa not found: ID does not exist" containerID="c08334efbf800b22adeda05a4abdc31c55173a07044b39d763b4fbf75ba357fa"
Nov 03 16:34:40.471090 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.471046    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08334efbf800b22adeda05a4abdc31c55173a07044b39d763b4fbf75ba357fa"} err="failed to get container status \"c08334efbf800b22adeda05a4abdc31c55173a07044b39d763b4fbf75ba357fa\": rpc error: code = NotFound desc = could not find container \"c08334efbf800b22adeda05a4abdc31c55173a07044b39d763b4fbf75ba357fa\": container with ID starting with c08334efbf800b22adeda05a4abdc31c55173a07044b39d763b4fbf75ba357fa not found: ID does not exist"
Nov 03 16:34:40.484381 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.484354    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"]
Nov 03 16:34:40.488943 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.488923    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ba8f54/simulator-single-main-ba8f54-kserve-7477d59b7b-596xv"]
Nov 03 16:34:40.527544 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:40.527526    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"]
Nov 03 16:34:40.529506 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:34:40.529478    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136e618c_25fb_4f4d_bdcd_13af4f832c34.slice/crio-02f694621f5ad6de68f9e15a1b3cfe449641f9648480ae9b9f6d893db04c01ff WatchSource:0}: Error finding container 02f694621f5ad6de68f9e15a1b3cfe449641f9648480ae9b9f6d893db04c01ff: Status 404 returned error can't find the container with id 02f694621f5ad6de68f9e15a1b3cfe449641f9648480ae9b9f6d893db04c01ff
Nov 03 16:34:41.469868 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:41.469837    6506 generic.go:356] "Generic (PLEG): container finished" podID="136e618c-25fb-4f4d-bdcd-13af4f832c34" containerID="9f92ca61469f076d2fa4e361c0794a34fb2ee2ed21d1e9783a3116dc05be0a30" exitCode=0
Nov 03 16:34:41.470260 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:41.469873    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl" event={"ID":"136e618c-25fb-4f4d-bdcd-13af4f832c34","Type":"ContainerDied","Data":"9f92ca61469f076d2fa4e361c0794a34fb2ee2ed21d1e9783a3116dc05be0a30"}
Nov 03 16:34:41.470260 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:41.469909    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl" event={"ID":"136e618c-25fb-4f4d-bdcd-13af4f832c34","Type":"ContainerStarted","Data":"02f694621f5ad6de68f9e15a1b3cfe449641f9648480ae9b9f6d893db04c01ff"}
Nov 03 16:34:42.102141 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:42.102108    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e84614a-8475-431c-a256-df6c8006deaf" path="/var/lib/kubelet/pods/2e84614a-8475-431c-a256-df6c8006deaf/volumes"
Nov 03 16:34:42.607825 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:42.607786    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"
Nov 03 16:34:42.739945 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:42.739868    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136e618c-25fb-4f4d-bdcd-13af4f832c34-kube-api-access-pfxf9\" (UniqueName: \"kubernetes.io/projected/136e618c-25fb-4f4d-bdcd-13af4f832c34-kube-api-access-pfxf9\") pod \"136e618c-25fb-4f4d-bdcd-13af4f832c34\" (UID: \"136e618c-25fb-4f4d-bdcd-13af4f832c34\") "
Nov 03 16:34:42.739945 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:42.739919    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/136e618c-25fb-4f4d-bdcd-13af4f832c34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136e618c-25fb-4f4d-bdcd-13af4f832c34-maas-api-xks-ca\") pod \"136e618c-25fb-4f4d-bdcd-13af4f832c34\" (UID: \"136e618c-25fb-4f4d-bdcd-13af4f832c34\") "
Nov 03 16:34:42.740157 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:42.740039    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136e618c-25fb-4f4d-bdcd-13af4f832c34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136e618c-25fb-4f4d-bdcd-13af4f832c34-maas-api-openshift-ca\") pod \"136e618c-25fb-4f4d-bdcd-13af4f832c34\" (UID: \"136e618c-25fb-4f4d-bdcd-13af4f832c34\") "
Nov 03 16:34:42.740406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:42.740387    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136e618c-25fb-4f4d-bdcd-13af4f832c34-maas-api-openshift-ca" pod "136e618c-25fb-4f4d-bdcd-13af4f832c34" (UID: "136e618c-25fb-4f4d-bdcd-13af4f832c34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:34:42.742090 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:42.742061    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/136e618c-25fb-4f4d-bdcd-13af4f832c34-maas-api-xks-ca" pod "136e618c-25fb-4f4d-bdcd-13af4f832c34" (UID: "136e618c-25fb-4f4d-bdcd-13af4f832c34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:34:42.742179 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:42.742131    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136e618c-25fb-4f4d-bdcd-13af4f832c34-kube-api-access-pfxf9" pod "136e618c-25fb-4f4d-bdcd-13af4f832c34" (UID: "136e618c-25fb-4f4d-bdcd-13af4f832c34"). InnerVolumeSpecName "kube-api-access-pfxf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:34:42.841107 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:42.841083    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfxf9\" (UniqueName: \"kubernetes.io/projected/136e618c-25fb-4f4d-bdcd-13af4f832c34-kube-api-access-pfxf9\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:42.841107 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:42.841104    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136e618c-25fb-4f4d-bdcd-13af4f832c34-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:42.841263 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:42.841120    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136e618c-25fb-4f4d-bdcd-13af4f832c34-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:43.481255 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:43.481226    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"
Nov 03 16:34:43.481448 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:43.481271    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl" event={"ID":"136e618c-25fb-4f4d-bdcd-13af4f832c34","Type":"ContainerDied","Data":"02f694621f5ad6de68f9e15a1b3cfe449641f9648480ae9b9f6d893db04c01ff"}
Nov 03 16:34:43.481448 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:34:43.481325    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02f694621f5ad6de68f9e15a1b3cfe449641f9648480ae9b9f6d893db04c01ff"
Nov 03 16:34:58.795498 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:34:58.795414    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:35:01.599434 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:01.599401    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:35:02.615721 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:02.615684    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:35:08.082271 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.082238    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"]
Nov 03 16:35:08.082864 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.082701    6506 cpu_manager.go:414] "containerMap: removing container" podUID="2e84614a-8475-431c-a256-df6c8006deaf" containerName="main"
Nov 03 16:35:08.082864 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.082719    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e84614a-8475-431c-a256-df6c8006deaf" containerName="main"
Nov 03 16:35:08.082864 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.082821    6506 cpu_manager.go:414] "containerMap: removing container" podUID="136e618c-25fb-4f4d-bdcd-13af4f832c34" containerName="revoke-keys"
Nov 03 16:35:08.082864 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.082833    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136e618c-25fb-4f4d-bdcd-13af4f832c34" containerName="revoke-keys"
Nov 03 16:35:08.083209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.082953    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="136e618c-25fb-4f4d-bdcd-13af4f832c34" containerName="revoke-keys"
Nov 03 16:35:08.083209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.082976    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e84614a-8475-431c-a256-df6c8006deaf" containerName="main"
Nov 03 16:35:08.087513 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.087496    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.105501 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.105473    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"]
Nov 03 16:35:08.262936 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.262904    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hdqc\" (UniqueName: \"kubernetes.io/projected/da47afac-7b95-44ce-867c-1610ca16c13a-kube-api-access-2hdqc\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.263092 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.262956    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da47afac-7b95-44ce-867c-1610ca16c13a-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.263092 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.263053    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da47afac-7b95-44ce-867c-1610ca16c13a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.263092 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.263083    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da47afac-7b95-44ce-867c-1610ca16c13a-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.263281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.263201    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da47afac-7b95-44ce-867c-1610ca16c13a-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.263281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.263233    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47afac-7b95-44ce-867c-1610ca16c13a-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.263281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.263259    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da47afac-7b95-44ce-867c-1610ca16c13a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.263411 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.263291    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47afac-7b95-44ce-867c-1610ca16c13a-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.263411 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.263334    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da47afac-7b95-44ce-867c-1610ca16c13a-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.364139 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.364059    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da47afac-7b95-44ce-867c-1610ca16c13a-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.364139 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.364100    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47afac-7b95-44ce-867c-1610ca16c13a-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.364139 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.364122    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da47afac-7b95-44ce-867c-1610ca16c13a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.364399 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.364152    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47afac-7b95-44ce-867c-1610ca16c13a-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.364399 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.364177    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da47afac-7b95-44ce-867c-1610ca16c13a-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.364399 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.364244    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hdqc\" (UniqueName: \"kubernetes.io/projected/da47afac-7b95-44ce-867c-1610ca16c13a-kube-api-access-2hdqc\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.364399 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.364279    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da47afac-7b95-44ce-867c-1610ca16c13a-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.364399 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.364308    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da47afac-7b95-44ce-867c-1610ca16c13a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.364399 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.364392    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da47afac-7b95-44ce-867c-1610ca16c13a-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.364702 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.364516    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da47afac-7b95-44ce-867c-1610ca16c13a-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.364837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.364776    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47afac-7b95-44ce-867c-1610ca16c13a-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.364920 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.364885    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47afac-7b95-44ce-867c-1610ca16c13a-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.364969 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.364943    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da47afac-7b95-44ce-867c-1610ca16c13a-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.365241 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.365221    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da47afac-7b95-44ce-867c-1610ca16c13a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.366598 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.366579    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da47afac-7b95-44ce-867c-1610ca16c13a-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.366824 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.366789    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da47afac-7b95-44ce-867c-1610ca16c13a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.378517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.378493    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hdqc\" (UniqueName: \"kubernetes.io/projected/da47afac-7b95-44ce-867c-1610ca16c13a-kube-api-access-2hdqc\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.378689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.378590    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da47afac-7b95-44ce-867c-1610ca16c13a-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z5jhm\" (UID: \"da47afac-7b95-44ce-867c-1610ca16c13a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.401177 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.401157    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:08.664688 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.664599    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:35:08.685377 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.685354    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"]
Nov 03 16:35:08.686829 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:35:08.686790    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda47afac_7b95_44ce_867c_1610ca16c13a.slice/crio-e98b3a9c442a55187012748cf01e0f9a632fc4226dcdeb17d2c713636a3a4d09 WatchSource:0}: Error finding container e98b3a9c442a55187012748cf01e0f9a632fc4226dcdeb17d2c713636a3a4d09: Status 404 returned error can't find the container with id e98b3a9c442a55187012748cf01e0f9a632fc4226dcdeb17d2c713636a3a4d09
Nov 03 16:35:08.688946 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.688916    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:35:08.689013 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.688983    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:35:08.689051 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:08.689011    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:35:09.612751 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:09.612709    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" event={"ID":"da47afac-7b95-44ce-867c-1610ca16c13a","Type":"ContainerStarted","Data":"6698d2a950c88ba99645331aee233ceed722ae925077a48e8b617cd77636e308"}
Nov 03 16:35:09.613318 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:09.612779    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" event={"ID":"da47afac-7b95-44ce-867c-1610ca16c13a","Type":"ContainerStarted","Data":"e98b3a9c442a55187012748cf01e0f9a632fc4226dcdeb17d2c713636a3a4d09"}
Nov 03 16:35:09.640606 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:09.640556    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" podStartSLOduration=1.640544156 podStartE2EDuration="1.640544156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:09.638942566 +0000 UTC m=+1923.652340703" watchObservedRunningTime="2026-11-03 16:35:09.640544156 +0000 UTC m=+1923.653942302"
Nov 03 16:35:10.402277 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:10.402242    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:10.407289 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:10.407258    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:10.448560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:10.448532    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:35:10.617467 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:10.617433    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:10.618178 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:10.618149    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:35:22.181158 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:22.181126    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:35:23.047094 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.047053    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"]
Nov 03 16:35:23.054219 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.054177    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.067455 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.067426    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"]
Nov 03 16:35:23.094429 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.094386    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf76626-1d92-4255-8343-5ae7f38a01ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.094594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.094439    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf76626-1d92-4255-8343-5ae7f38a01ab-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.094594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.094478    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf76626-1d92-4255-8343-5ae7f38a01ab-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.094594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.094544    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf76626-1d92-4255-8343-5ae7f38a01ab-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.094594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.094579    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k5fd\" (UniqueName: \"kubernetes.io/projected/2cf76626-1d92-4255-8343-5ae7f38a01ab-kube-api-access-7k5fd\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.094888 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.094635    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf76626-1d92-4255-8343-5ae7f38a01ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.094888 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.094675    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf76626-1d92-4255-8343-5ae7f38a01ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.094888 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.094709    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf76626-1d92-4255-8343-5ae7f38a01ab-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.094888 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.094729    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf76626-1d92-4255-8343-5ae7f38a01ab-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.195908 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.195872    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf76626-1d92-4255-8343-5ae7f38a01ab-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.196346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.195911    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf76626-1d92-4255-8343-5ae7f38a01ab-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.196346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.195960    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf76626-1d92-4255-8343-5ae7f38a01ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.196346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.195992    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf76626-1d92-4255-8343-5ae7f38a01ab-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.196346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.196029    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf76626-1d92-4255-8343-5ae7f38a01ab-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.196346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.196127    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf76626-1d92-4255-8343-5ae7f38a01ab-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.196346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.196165    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k5fd\" (UniqueName: \"kubernetes.io/projected/2cf76626-1d92-4255-8343-5ae7f38a01ab-kube-api-access-7k5fd\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.196346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.196226    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf76626-1d92-4255-8343-5ae7f38a01ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.196346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.196276    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf76626-1d92-4255-8343-5ae7f38a01ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.196346 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.196318    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf76626-1d92-4255-8343-5ae7f38a01ab-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.196869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.196440    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf76626-1d92-4255-8343-5ae7f38a01ab-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.196869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.196706    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf76626-1d92-4255-8343-5ae7f38a01ab-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.196976 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.196958    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf76626-1d92-4255-8343-5ae7f38a01ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.197018 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.196969    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf76626-1d92-4255-8343-5ae7f38a01ab-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.198450 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.198429    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf76626-1d92-4255-8343-5ae7f38a01ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.198927 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.198898    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf76626-1d92-4255-8343-5ae7f38a01ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.209221 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.209193    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf76626-1d92-4255-8343-5ae7f38a01ab-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.209418 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.209397    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k5fd\" (UniqueName: \"kubernetes.io/projected/2cf76626-1d92-4255-8343-5ae7f38a01ab-kube-api-access-7k5fd\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-9z9vz\" (UID: \"2cf76626-1d92-4255-8343-5ae7f38a01ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.368052 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.367968    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:23.651626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.651600    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"]
Nov 03 16:35:23.652766 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:35:23.652735    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf76626_1d92_4255_8343_5ae7f38a01ab.slice/crio-4b10699a4b3a7b45d2aa94dcc67f2bbdc0ef6be9dc9d5c54d12faca7e8c609ef WatchSource:0}: Error finding container 4b10699a4b3a7b45d2aa94dcc67f2bbdc0ef6be9dc9d5c54d12faca7e8c609ef: Status 404 returned error can't find the container with id 4b10699a4b3a7b45d2aa94dcc67f2bbdc0ef6be9dc9d5c54d12faca7e8c609ef
Nov 03 16:35:23.655164 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.655136    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:35:23.655243 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.655196    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:35:23.655243 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.655225    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:35:23.679620 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:23.679592    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" event={"ID":"2cf76626-1d92-4255-8343-5ae7f38a01ab","Type":"ContainerStarted","Data":"4b10699a4b3a7b45d2aa94dcc67f2bbdc0ef6be9dc9d5c54d12faca7e8c609ef"}
Nov 03 16:35:24.685861 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:24.685827    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" event={"ID":"2cf76626-1d92-4255-8343-5ae7f38a01ab","Type":"ContainerStarted","Data":"d3d1a607242cd26fe74b7f1dec08c8f65d40b4a968ba6baf64ceccc08e7ad19f"}
Nov 03 16:35:24.713614 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:24.713572    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" podStartSLOduration=1.7135584750000001 podStartE2EDuration="1.713558475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:24.711570273 +0000 UTC m=+1938.724968421" watchObservedRunningTime="2026-11-03 16:35:24.713558475 +0000 UTC m=+1938.726956693"
Nov 03 16:35:25.368901 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:25.368873    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:25.373762 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:25.373735    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:25.691120 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:25.691036    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:25.691987 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:25.691972    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:35:28.798171 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:35:28.798129    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:35:47.342166 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:47.342131    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:35:47.502484 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:35:47.502444    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:35:58.795582 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:35:58.795543    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:36:10.522444 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:10.522376    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:36:22.989426 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:22.989327    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"]
Nov 03 16:36:22.993166 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:22.993144    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"
Nov 03 16:36:22.997342 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:22.997325    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:36:22.998510 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:22.998489    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:36:22.998629 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:22.998572    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:36:22.998629 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:22.998594    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:36:23.003966 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:23.003945    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"]
Nov 03 16:36:23.085452 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:23.085421    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea26cba-bb06-47e0-8247-30e61f2a5a36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-shflt\" (UID: \"7ea26cba-bb06-47e0-8247-30e61f2a5a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"
Nov 03 16:36:23.085666 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:23.085652    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzj97\" (UniqueName: \"kubernetes.io/projected/7ea26cba-bb06-47e0-8247-30e61f2a5a36-kube-api-access-pzj97\") pod \"maas-api-revoke-sub-d576e68b-shflt\" (UID: \"7ea26cba-bb06-47e0-8247-30e61f2a5a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"
Nov 03 16:36:23.085865 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:23.085854    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea26cba-bb06-47e0-8247-30e61f2a5a36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-shflt\" (UID: \"7ea26cba-bb06-47e0-8247-30e61f2a5a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"
Nov 03 16:36:23.187264 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:23.187223    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea26cba-bb06-47e0-8247-30e61f2a5a36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-shflt\" (UID: \"7ea26cba-bb06-47e0-8247-30e61f2a5a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"
Nov 03 16:36:23.187443 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:23.187279    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea26cba-bb06-47e0-8247-30e61f2a5a36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-shflt\" (UID: \"7ea26cba-bb06-47e0-8247-30e61f2a5a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"
Nov 03 16:36:23.187443 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:23.187300    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzj97\" (UniqueName: \"kubernetes.io/projected/7ea26cba-bb06-47e0-8247-30e61f2a5a36-kube-api-access-pzj97\") pod \"maas-api-revoke-sub-d576e68b-shflt\" (UID: \"7ea26cba-bb06-47e0-8247-30e61f2a5a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"
Nov 03 16:36:23.188317 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:23.188294    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea26cba-bb06-47e0-8247-30e61f2a5a36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-shflt\" (UID: \"7ea26cba-bb06-47e0-8247-30e61f2a5a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"
Nov 03 16:36:23.190322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:23.190297    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea26cba-bb06-47e0-8247-30e61f2a5a36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-shflt\" (UID: \"7ea26cba-bb06-47e0-8247-30e61f2a5a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"
Nov 03 16:36:23.199835 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:23.199812    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzj97\" (UniqueName: \"kubernetes.io/projected/7ea26cba-bb06-47e0-8247-30e61f2a5a36-kube-api-access-pzj97\") pod \"maas-api-revoke-sub-d576e68b-shflt\" (UID: \"7ea26cba-bb06-47e0-8247-30e61f2a5a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"
Nov 03 16:36:23.327822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:23.326001    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"
Nov 03 16:36:23.655144 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:23.655060    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"]
Nov 03 16:36:23.662823 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:36:23.662601    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea26cba_bb06_47e0_8247_30e61f2a5a36.slice/crio-1f1d85c01f87bc32c03cfe87346a58bd86b5d223d88a13f1c48a85dc6cda8183 WatchSource:0}: Error finding container 1f1d85c01f87bc32c03cfe87346a58bd86b5d223d88a13f1c48a85dc6cda8183: Status 404 returned error can't find the container with id 1f1d85c01f87bc32c03cfe87346a58bd86b5d223d88a13f1c48a85dc6cda8183
Nov 03 16:36:24.031157 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:24.031074    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt" event={"ID":"7ea26cba-bb06-47e0-8247-30e61f2a5a36","Type":"ContainerStarted","Data":"1f1d85c01f87bc32c03cfe87346a58bd86b5d223d88a13f1c48a85dc6cda8183"}
Nov 03 16:36:25.038147 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:25.038104    6506 generic.go:356] "Generic (PLEG): container finished" podID="7ea26cba-bb06-47e0-8247-30e61f2a5a36" containerID="9db2cf5877a0302f7247fb3110bf5a9cd861b5742dd425f13a446b8f72969cc8" exitCode=0
Nov 03 16:36:25.038684 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:25.038233    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt" event={"ID":"7ea26cba-bb06-47e0-8247-30e61f2a5a36","Type":"ContainerDied","Data":"9db2cf5877a0302f7247fb3110bf5a9cd861b5742dd425f13a446b8f72969cc8"}
Nov 03 16:36:26.198539 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:26.198507    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"
Nov 03 16:36:26.318217 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:26.318124    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ea26cba-bb06-47e0-8247-30e61f2a5a36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea26cba-bb06-47e0-8247-30e61f2a5a36-maas-api-xks-ca\") pod \"7ea26cba-bb06-47e0-8247-30e61f2a5a36\" (UID: \"7ea26cba-bb06-47e0-8247-30e61f2a5a36\") "
Nov 03 16:36:26.318399 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:26.318240    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea26cba-bb06-47e0-8247-30e61f2a5a36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea26cba-bb06-47e0-8247-30e61f2a5a36-maas-api-openshift-ca\") pod \"7ea26cba-bb06-47e0-8247-30e61f2a5a36\" (UID: \"7ea26cba-bb06-47e0-8247-30e61f2a5a36\") "
Nov 03 16:36:26.318399 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:26.318299    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea26cba-bb06-47e0-8247-30e61f2a5a36-kube-api-access-pzj97\" (UniqueName: \"kubernetes.io/projected/7ea26cba-bb06-47e0-8247-30e61f2a5a36-kube-api-access-pzj97\") pod \"7ea26cba-bb06-47e0-8247-30e61f2a5a36\" (UID: \"7ea26cba-bb06-47e0-8247-30e61f2a5a36\") "
Nov 03 16:36:26.318761 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:26.318628    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea26cba-bb06-47e0-8247-30e61f2a5a36-maas-api-openshift-ca" pod "7ea26cba-bb06-47e0-8247-30e61f2a5a36" (UID: "7ea26cba-bb06-47e0-8247-30e61f2a5a36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:36:26.320959 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:26.320900    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea26cba-bb06-47e0-8247-30e61f2a5a36-maas-api-xks-ca" pod "7ea26cba-bb06-47e0-8247-30e61f2a5a36" (UID: "7ea26cba-bb06-47e0-8247-30e61f2a5a36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:36:26.321079 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:26.321022    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea26cba-bb06-47e0-8247-30e61f2a5a36-kube-api-access-pzj97" pod "7ea26cba-bb06-47e0-8247-30e61f2a5a36" (UID: "7ea26cba-bb06-47e0-8247-30e61f2a5a36"). InnerVolumeSpecName "kube-api-access-pzj97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:36:26.419628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:26.419582    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea26cba-bb06-47e0-8247-30e61f2a5a36-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:36:26.419628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:26.419622    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzj97\" (UniqueName: \"kubernetes.io/projected/7ea26cba-bb06-47e0-8247-30e61f2a5a36-kube-api-access-pzj97\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:36:26.419628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:26.419639    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea26cba-bb06-47e0-8247-30e61f2a5a36-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:36:26.661035 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:26.660950    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:26.661035 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:26.661013    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" podUID="da47afac-7b95-44ce-867c-1610ca16c13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:26.733980 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:26.733932    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:26.734175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:26.733996    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" podUID="2cf76626-1d92-4255-8343-5ae7f38a01ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:27.051943 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:27.051907    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"
Nov 03 16:36:27.052124 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:27.051906    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt" event={"ID":"7ea26cba-bb06-47e0-8247-30e61f2a5a36","Type":"ContainerDied","Data":"1f1d85c01f87bc32c03cfe87346a58bd86b5d223d88a13f1c48a85dc6cda8183"}
Nov 03 16:36:27.052124 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:27.052043    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f1d85c01f87bc32c03cfe87346a58bd86b5d223d88a13f1c48a85dc6cda8183"
Nov 03 16:36:28.797171 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:36:28.797135    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:36:29.063647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:29.063565    6506 generic.go:356] "Generic (PLEG): container finished" podID="2cf76626-1d92-4255-8343-5ae7f38a01ab" containerID="d3d1a607242cd26fe74b7f1dec08c8f65d40b4a968ba6baf64ceccc08e7ad19f" exitCode=0
Nov 03 16:36:29.063788 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:29.063639    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" event={"ID":"2cf76626-1d92-4255-8343-5ae7f38a01ab","Type":"ContainerDied","Data":"d3d1a607242cd26fe74b7f1dec08c8f65d40b4a968ba6baf64ceccc08e7ad19f"}
Nov 03 16:36:29.064058 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:29.064038    6506 scope.go:122] "RemoveContainer" containerID="d3d1a607242cd26fe74b7f1dec08c8f65d40b4a968ba6baf64ceccc08e7ad19f"
Nov 03 16:36:29.065290 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:29.065261    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:36:29.065367 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:29.065337    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:36:29.065367 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:29.065357    6506 generic.go:356] "Generic (PLEG): container finished" podID="da47afac-7b95-44ce-867c-1610ca16c13a" containerID="6698d2a950c88ba99645331aee233ceed722ae925077a48e8b617cd77636e308" exitCode=0
Nov 03 16:36:29.065435 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:29.065376    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:36:29.065435 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:29.065393    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" event={"ID":"da47afac-7b95-44ce-867c-1610ca16c13a","Type":"ContainerDied","Data":"6698d2a950c88ba99645331aee233ceed722ae925077a48e8b617cd77636e308"}
Nov 03 16:36:29.065676 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:29.065663    6506 scope.go:122] "RemoveContainer" containerID="6698d2a950c88ba99645331aee233ceed722ae925077a48e8b617cd77636e308"
Nov 03 16:36:29.066667 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:29.066640    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:36:29.066783 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:29.066707    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:36:29.066783 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:29.066742    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:36:30.072355 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:30.072318    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" event={"ID":"2cf76626-1d92-4255-8343-5ae7f38a01ab","Type":"ContainerStarted","Data":"41a1663053c28d14d9eec77380373e90b7fa873b57fa74d2ec28b0aebe9299da"}
Nov 03 16:36:30.074315 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:30.074290    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" event={"ID":"da47afac-7b95-44ce-867c-1610ca16c13a","Type":"ContainerStarted","Data":"b1ad4b78e1d81f003ee8a2170e36a783412a5490905bf86f70ba3f5207b85284"}
Nov 03 16:36:30.368779 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:30.368694    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:36:30.368961 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:30.368901    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 16:36:30.368961 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:30.368943    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" podUID="2cf76626-1d92-4255-8343-5ae7f38a01ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 16:36:30.402104 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:30.402073    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:36:30.402271 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:30.402244    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 16:36:30.402323 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:30.402277    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" podUID="da47afac-7b95-44ce-867c-1610ca16c13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 16:36:31.368661 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:31.368627    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 16:36:31.369071 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:31.368677    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" podUID="2cf76626-1d92-4255-8343-5ae7f38a01ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 16:36:31.401947 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:31.401923    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 16:36:31.402071 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:31.401975    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" podUID="da47afac-7b95-44ce-867c-1610ca16c13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 16:36:32.368526 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:32.368488    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 16:36:32.368727 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:32.368544    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" podUID="2cf76626-1d92-4255-8343-5ae7f38a01ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 16:36:32.401903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:32.401868    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 16:36:32.402057 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:32.401922    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" podUID="da47afac-7b95-44ce-867c-1610ca16c13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 16:36:33.369408 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:33.369371    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 16:36:33.369848 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:33.369439    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" podUID="2cf76626-1d92-4255-8343-5ae7f38a01ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 16:36:33.401869 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:33.401839    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 16:36:33.402017 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:33.401887    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" podUID="da47afac-7b95-44ce-867c-1610ca16c13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 16:36:34.369362 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:34.369322    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 16:36:34.369538 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:34.369379    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" podUID="2cf76626-1d92-4255-8343-5ae7f38a01ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 16:36:34.401837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:34.401780    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 16:36:34.401988 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:34.401852    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" podUID="da47afac-7b95-44ce-867c-1610ca16c13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 16:36:35.368734 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:35.368692    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 16:36:35.368940 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:35.368758    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" podUID="2cf76626-1d92-4255-8343-5ae7f38a01ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 16:36:35.401732 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:35.401698    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 16:36:35.402151 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:35.401760    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" podUID="da47afac-7b95-44ce-867c-1610ca16c13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 16:36:36.368837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:36.368779    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 16:36:36.369073 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:36.368864    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" podUID="2cf76626-1d92-4255-8343-5ae7f38a01ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 16:36:36.402576 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:36.402546    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 16:36:36.402959 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:36.402597    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" podUID="da47afac-7b95-44ce-867c-1610ca16c13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 16:36:38.369633 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:38.369558    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:38.370306 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:38.370255    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" podUID="2cf76626-1d92-4255-8343-5ae7f38a01ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:38.443975 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:38.443926    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:38.444198 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:38.444019    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" podUID="da47afac-7b95-44ce-867c-1610ca16c13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:39.414951 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:39.414912    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:39.415561 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:39.415529    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" podUID="2cf76626-1d92-4255-8343-5ae7f38a01ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:39.487999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:39.487958    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:39.488186 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:39.488015    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" podUID="da47afac-7b95-44ce-867c-1610ca16c13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:40.416441 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:40.416402    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:40.416944 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:40.416464    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" podUID="2cf76626-1d92-4255-8343-5ae7f38a01ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:40.488448 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:40.488414    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:40.488631 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:40.488489    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" podUID="da47afac-7b95-44ce-867c-1610ca16c13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:40.618054 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:40.618021    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:36:40.691918 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:40.691845    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:36:41.464772 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:41.462961    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:41.464772 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:41.463043    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" podUID="2cf76626-1d92-4255-8343-5ae7f38a01ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:41.529993 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:41.529940    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:41.530198 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:41.530006    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm" podUID="da47afac-7b95-44ce-867c-1610ca16c13a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:42.166578 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.166546    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"]
Nov 03 16:36:42.167920 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.167367    6506 cpu_manager.go:414] "containerMap: removing container" podUID="7ea26cba-bb06-47e0-8247-30e61f2a5a36" containerName="revoke-keys"
Nov 03 16:36:42.168095 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.168076    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea26cba-bb06-47e0-8247-30e61f2a5a36" containerName="revoke-keys"
Nov 03 16:36:42.168324 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.168312    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea26cba-bb06-47e0-8247-30e61f2a5a36" containerName="revoke-keys"
Nov 03 16:36:42.172206 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.172185    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"
Nov 03 16:36:42.179115 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.179090    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:36:42.180389 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.180300    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:36:42.180389 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.180382    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:36:42.180554 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.180409    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:36:42.182896 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.182875    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"]
Nov 03 16:36:42.205231 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.205205    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wnhbh\" (UID: \"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"
Nov 03 16:36:42.205437 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.205411    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wnhbh\" (UID: \"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"
Nov 03 16:36:42.205512 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.205484    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v8fw\" (UniqueName: \"kubernetes.io/projected/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-kube-api-access-5v8fw\") pod \"maas-api-revoke-sub-d2bc93bc-wnhbh\" (UID: \"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"
Nov 03 16:36:42.306435 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.306391    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wnhbh\" (UID: \"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"
Nov 03 16:36:42.306435 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.306435    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wnhbh\" (UID: \"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"
Nov 03 16:36:42.306709 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.306474    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8fw\" (UniqueName: \"kubernetes.io/projected/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-kube-api-access-5v8fw\") pod \"maas-api-revoke-sub-d2bc93bc-wnhbh\" (UID: \"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"
Nov 03 16:36:42.307220 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.307179    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wnhbh\" (UID: \"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"
Nov 03 16:36:42.309484 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.309440    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wnhbh\" (UID: \"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"
Nov 03 16:36:42.324433 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.324402    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8fw\" (UniqueName: \"kubernetes.io/projected/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-kube-api-access-5v8fw\") pod \"maas-api-revoke-sub-d2bc93bc-wnhbh\" (UID: \"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"
Nov 03 16:36:42.488561 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.488236    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"
Nov 03 16:36:42.489414 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.489386    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:36:42.505972 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.505945    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:42.506089 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.506011    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz" podUID="2cf76626-1d92-4255-8343-5ae7f38a01ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:42.506908 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.506886    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z5jhm"
Nov 03 16:36:42.668991 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.668964    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:36:42.669889 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.669870    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9z9vz"
Nov 03 16:36:42.782110 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:42.782085    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"]
Nov 03 16:36:42.784619 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:36:42.784591    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ee3fd3_6307_4b57_92aa_9ac25f13ee7b.slice/crio-1a0e7c5acb0467f41c8857d8457988ab6e1d8102282b3d98447e0d62e7330081 WatchSource:0}: Error finding container 1a0e7c5acb0467f41c8857d8457988ab6e1d8102282b3d98447e0d62e7330081: Status 404 returned error can't find the container with id 1a0e7c5acb0467f41c8857d8457988ab6e1d8102282b3d98447e0d62e7330081
Nov 03 16:36:43.178585 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:43.178544    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh" event={"ID":"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b","Type":"ContainerStarted","Data":"3fcfe1d6270c8d31986ab7f91a6bd567753c6a45b02933862ab3152b95e913ef"}
Nov 03 16:36:43.178830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:43.178607    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh" event={"ID":"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b","Type":"ContainerStarted","Data":"1a0e7c5acb0467f41c8857d8457988ab6e1d8102282b3d98447e0d62e7330081"}
Nov 03 16:36:43.197599 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:43.197548    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh" podStartSLOduration=1.197536442 podStartE2EDuration="1.197536442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:36:43.196393798 +0000 UTC m=+2017.209791945" watchObservedRunningTime="2026-11-03 16:36:43.197536442 +0000 UTC m=+2017.210934773"
Nov 03 16:36:44.185415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:44.185378    6506 generic.go:356] "Generic (PLEG): container finished" podID="f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b" containerID="3fcfe1d6270c8d31986ab7f91a6bd567753c6a45b02933862ab3152b95e913ef" exitCode=0
Nov 03 16:36:44.185787 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:44.185451    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh" event={"ID":"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b","Type":"ContainerDied","Data":"3fcfe1d6270c8d31986ab7f91a6bd567753c6a45b02933862ab3152b95e913ef"}
Nov 03 16:36:45.317616 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:45.317591    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"
Nov 03 16:36:45.435604 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:45.435574    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-kube-api-access-5v8fw\" (UniqueName: \"kubernetes.io/projected/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-kube-api-access-5v8fw\") pod \"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b\" (UID: \"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b\") "
Nov 03 16:36:45.435777 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:45.435633    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-maas-api-xks-ca\") pod \"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b\" (UID: \"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b\") "
Nov 03 16:36:45.435777 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:45.435696    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-maas-api-openshift-ca\") pod \"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b\" (UID: \"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b\") "
Nov 03 16:36:45.436121 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:45.436088    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-maas-api-openshift-ca" pod "f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b" (UID: "f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:36:45.437861 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:45.437839    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-maas-api-xks-ca" pod "f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b" (UID: "f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:36:45.437947 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:45.437926    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-kube-api-access-5v8fw" pod "f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b" (UID: "f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b"). InnerVolumeSpecName "kube-api-access-5v8fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:36:45.536535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:45.536507    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:36:45.536535 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:45.536532    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v8fw\" (UniqueName: \"kubernetes.io/projected/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-kube-api-access-5v8fw\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:36:45.536700 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:45.536542    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:36:46.196364 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:46.196335    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"
Nov 03 16:36:46.196364 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:46.196352    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh" event={"ID":"f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b","Type":"ContainerDied","Data":"1a0e7c5acb0467f41c8857d8457988ab6e1d8102282b3d98447e0d62e7330081"}
Nov 03 16:36:46.196578 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:46.196399    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a0e7c5acb0467f41c8857d8457988ab6e1d8102282b3d98447e0d62e7330081"
Nov 03 16:36:53.181956 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.181920    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"]
Nov 03 16:36:53.182517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.182497    6506 cpu_manager.go:414] "containerMap: removing container" podUID="f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b" containerName="revoke-keys"
Nov 03 16:36:53.182601 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.182521    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b" containerName="revoke-keys"
Nov 03 16:36:53.182656 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.182624    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b" containerName="revoke-keys"
Nov 03 16:36:53.185369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.185347    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.203168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.203141    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"]
Nov 03 16:36:53.306725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.306691    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b93c18-e065-4198-897d-2d7dcdcc8d07-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.306725 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.306725    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b93c18-e065-4198-897d-2d7dcdcc8d07-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.306997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.306746    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b93c18-e065-4198-897d-2d7dcdcc8d07-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.306997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.306876    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b93c18-e065-4198-897d-2d7dcdcc8d07-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.306997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.306914    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b93c18-e065-4198-897d-2d7dcdcc8d07-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.306997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.306943    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh4sk\" (UniqueName: \"kubernetes.io/projected/f9b93c18-e065-4198-897d-2d7dcdcc8d07-kube-api-access-nh4sk\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.307162 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.307020    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b93c18-e065-4198-897d-2d7dcdcc8d07-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.307162 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.307051    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b93c18-e065-4198-897d-2d7dcdcc8d07-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.307162 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.307066    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b93c18-e065-4198-897d-2d7dcdcc8d07-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.408080 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.408043    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b93c18-e065-4198-897d-2d7dcdcc8d07-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.408080 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.408083    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b93c18-e065-4198-897d-2d7dcdcc8d07-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.408437 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.408106    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b93c18-e065-4198-897d-2d7dcdcc8d07-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.408437 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.408306    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b93c18-e065-4198-897d-2d7dcdcc8d07-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.408437 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.408402    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b93c18-e065-4198-897d-2d7dcdcc8d07-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.408695 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.408532    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b93c18-e065-4198-897d-2d7dcdcc8d07-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.408695 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.408547    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh4sk\" (UniqueName: \"kubernetes.io/projected/f9b93c18-e065-4198-897d-2d7dcdcc8d07-kube-api-access-nh4sk\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.408866 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.408843    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b93c18-e065-4198-897d-2d7dcdcc8d07-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.409029 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.408932    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b93c18-e065-4198-897d-2d7dcdcc8d07-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.409029 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.408975    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b93c18-e065-4198-897d-2d7dcdcc8d07-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.409029 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.409023    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b93c18-e065-4198-897d-2d7dcdcc8d07-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.409029 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.409049    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b93c18-e065-4198-897d-2d7dcdcc8d07-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.409443 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.409250    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b93c18-e065-4198-897d-2d7dcdcc8d07-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.409443 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.409408    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b93c18-e065-4198-897d-2d7dcdcc8d07-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.411603 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.411572    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b93c18-e065-4198-897d-2d7dcdcc8d07-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.411836 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.411816    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b93c18-e065-4198-897d-2d7dcdcc8d07-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.421434 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.421410    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b93c18-e065-4198-897d-2d7dcdcc8d07-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.421881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.421859    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh4sk\" (UniqueName: \"kubernetes.io/projected/f9b93c18-e065-4198-897d-2d7dcdcc8d07-kube-api-access-nh4sk\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lmt2w\" (UID: \"f9b93c18-e065-4198-897d-2d7dcdcc8d07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.498034 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.497949    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:53.782844 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.782819    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"]
Nov 03 16:36:53.784626 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:36:53.784595    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b93c18_e065_4198_897d_2d7dcdcc8d07.slice/crio-f991c6b578da323a7d128a0a98660856ea736ec1888d697e25d71d60a8be9691 WatchSource:0}: Error finding container f991c6b578da323a7d128a0a98660856ea736ec1888d697e25d71d60a8be9691: Status 404 returned error can't find the container with id f991c6b578da323a7d128a0a98660856ea736ec1888d697e25d71d60a8be9691
Nov 03 16:36:53.786849 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.786784    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:36:53.786962 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.786904    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:36:53.786962 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.786956    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:36:53.971608 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:53.971566    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:36:54.239918 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.239827    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w" event={"ID":"f9b93c18-e065-4198-897d-2d7dcdcc8d07","Type":"ContainerStarted","Data":"c7d5377666ffa3fd25c688b60d704cbeb9e1f5039b577feb2287405fd62916b9"}
Nov 03 16:36:54.239918 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.239893    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w" event={"ID":"f9b93c18-e065-4198-897d-2d7dcdcc8d07","Type":"ContainerStarted","Data":"f991c6b578da323a7d128a0a98660856ea736ec1888d697e25d71d60a8be9691"}
Nov 03 16:36:54.273152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.273068    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w" podStartSLOduration=1.273054508 podStartE2EDuration="1.273054508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:36:54.2713056 +0000 UTC m=+2028.284703747" watchObservedRunningTime="2026-11-03 16:36:54.273054508 +0000 UTC m=+2028.286452659"
Nov 03 16:36:54.337822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.328071    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"]
Nov 03 16:36:54.343959 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.343653    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.352896 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.352590    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-13ba71-openshift-default-dockercfg-kz9n5\""
Nov 03 16:36:54.376629 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.376595    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"]
Nov 03 16:36:54.449064 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.448409    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-workload-socket\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.449064 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.448515    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-data\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.449064 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.448543    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-token\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.449064 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.448590    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-envoy\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.449064 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.448636    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm7mg\" (UniqueName: \"kubernetes.io/projected/a5b3f102-010a-4db5-a02a-b4b5074dc229-kube-api-access-rm7mg\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.449064 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.448666    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-workload-certs\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.449064 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.448691    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-credential-socket\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.449064 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.448755    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b3f102-010a-4db5-a02a-b4b5074dc229-istiod-ca-cert\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.449064 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.448787    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-podinfo\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.499054 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.498960    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:36:54.500604 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.500570    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:36:54.500731 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.500645    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 16:36:54.500731 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.500693    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w" podUID="f9b93c18-e065-4198-897d-2d7dcdcc8d07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 16:36:54.549522 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.549488    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-data\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.549522 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.549524    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-token\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.549829 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.549552    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-envoy\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.549829 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.549580    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm7mg\" (UniqueName: \"kubernetes.io/projected/a5b3f102-010a-4db5-a02a-b4b5074dc229-kube-api-access-rm7mg\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.549829 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.549609    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-workload-certs\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.549829 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.549632    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-credential-socket\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.549829 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.549783    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b3f102-010a-4db5-a02a-b4b5074dc229-istiod-ca-cert\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.550114 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.549862    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-podinfo\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.550114 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.549913    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-workload-socket\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.550114 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.550011    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-data\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.550114 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.550053    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-credential-socket\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.550248 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.550223    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-workload-socket\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.550334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.550302    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-workload-certs\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.550540 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.550518    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b3f102-010a-4db5-a02a-b4b5074dc229-istiod-ca-cert\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.552399 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.552380    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-envoy\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.552728 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.552705    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-podinfo\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.563349 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.563327    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-token\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.563446 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.563402    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm7mg\" (UniqueName: \"kubernetes.io/projected/a5b3f102-010a-4db5-a02a-b4b5074dc229-kube-api-access-rm7mg\") pod \"e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") " pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.662395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.662353    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:54.681140 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.681093    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:36:54.969334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.969308    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"]
Nov 03 16:36:54.971514 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:36:54.971481    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b3f102_010a_4db5_a02a_b4b5074dc229.slice/crio-4082e60d0c43faef9430778cceb3cb649b33867f29baaa701b9d4010270cc515 WatchSource:0}: Error finding container 4082e60d0c43faef9430778cceb3cb649b33867f29baaa701b9d4010270cc515: Status 404 returned error can't find the container with id 4082e60d0c43faef9430778cceb3cb649b33867f29baaa701b9d4010270cc515
Nov 03 16:36:54.973665 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.973631    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:36:54.973769 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.973706    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:36:54.973769 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:54.973747    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:36:55.252844 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:55.252709    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp" event={"ID":"a5b3f102-010a-4db5-a02a-b4b5074dc229","Type":"ContainerStarted","Data":"012476df8f646bc5c116d2ded1d27b59ce51816b25a8c33e86a89b861f8a37a9"}
Nov 03 16:36:55.252844 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:55.252778    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp" event={"ID":"a5b3f102-010a-4db5-a02a-b4b5074dc229","Type":"ContainerStarted","Data":"4082e60d0c43faef9430778cceb3cb649b33867f29baaa701b9d4010270cc515"}
Nov 03 16:36:55.498459 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:55.498423    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 16:36:55.498642 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:55.498485    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w" podUID="f9b93c18-e065-4198-897d-2d7dcdcc8d07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 16:36:55.663461 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:55.663420    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:56.498234 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:56.498204    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 16:36:56.498627 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:56.498266    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w" podUID="f9b93c18-e065-4198-897d-2d7dcdcc8d07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 16:36:56.668236 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:56.668206    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:56.695171 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:56.695098    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp" podStartSLOduration=2.695085911 podStartE2EDuration="2.695085911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:36:55.279812161 +0000 UTC m=+2029.293210307" watchObservedRunningTime="2026-11-03 16:36:56.695085911 +0000 UTC m=+2030.708484118"
Nov 03 16:36:57.263907 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:57.263880    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:57.264917 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:57.264894    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:36:57.498496 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:57.498465    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 16:36:57.498888 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:57.498528    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w" podUID="f9b93c18-e065-4198-897d-2d7dcdcc8d07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 16:36:57.592233 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:57.592201    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:36:57.747825 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:57.747761    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:36:58.498586 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:58.498554    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 16:36:58.499044 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:58.498613    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w" podUID="f9b93c18-e065-4198-897d-2d7dcdcc8d07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 16:36:58.797175 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:36:58.797138    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:36:58.879338 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:58.879303    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:36:58.930817 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:58.926426    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"]
Nov 03 16:36:58.939990 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:58.939877    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"
Nov 03 16:36:58.947458 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:58.947432    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"]
Nov 03 16:36:58.951432 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:58.951410    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-0f006328-dockercfg-n4fsp\""
Nov 03 16:36:59.033158 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:59.029156    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-13ba71\""
Nov 03 16:36:59.083322 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:59.083238    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:36:59.093247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:59.093225    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab68de1-044b-4670-8491-493fb1c775e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5\" (UID: \"8ab68de1-044b-4670-8491-493fb1c775e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"
Nov 03 16:36:59.093381 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:59.093365    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d4nf\" (UniqueName: \"kubernetes.io/projected/8ab68de1-044b-4670-8491-493fb1c775e7-kube-api-access-2d4nf\") pod \"payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5\" (UID: \"8ab68de1-044b-4670-8491-493fb1c775e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"
Nov 03 16:36:59.193972 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:59.193932    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d4nf\" (UniqueName: \"kubernetes.io/projected/8ab68de1-044b-4670-8491-493fb1c775e7-kube-api-access-2d4nf\") pod \"payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5\" (UID: \"8ab68de1-044b-4670-8491-493fb1c775e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"
Nov 03 16:36:59.194126 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:59.193995    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab68de1-044b-4670-8491-493fb1c775e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5\" (UID: \"8ab68de1-044b-4670-8491-493fb1c775e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"
Nov 03 16:36:59.194644 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:59.194624    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab68de1-044b-4670-8491-493fb1c775e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5\" (UID: \"8ab68de1-044b-4670-8491-493fb1c775e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"
Nov 03 16:36:59.231507 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:59.231474    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d4nf\" (UniqueName: \"kubernetes.io/projected/8ab68de1-044b-4670-8491-493fb1c775e7-kube-api-access-2d4nf\") pod \"payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5\" (UID: \"8ab68de1-044b-4670-8491-493fb1c775e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"
Nov 03 16:36:59.257899 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:59.257873    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"
Nov 03 16:36:59.499302 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:59.499264    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 16:36:59.499660 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:59.499324    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w" podUID="f9b93c18-e065-4198-897d-2d7dcdcc8d07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 16:36:59.566041 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:59.566018    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"]
Nov 03 16:36:59.567649 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:36:59.567620    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab68de1_044b_4670_8491_493fb1c775e7.slice/crio-85478a5e89780398d136b009535eac6837e0636c6a5966fa03ed9007c06931dc WatchSource:0}: Error finding container 85478a5e89780398d136b009535eac6837e0636c6a5966fa03ed9007c06931dc: Status 404 returned error can't find the container with id 85478a5e89780398d136b009535eac6837e0636c6a5966fa03ed9007c06931dc
Nov 03 16:36:59.870440 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:36:59.870350    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:00.283531 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:00.283495    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5" event={"ID":"8ab68de1-044b-4670-8491-493fb1c775e7","Type":"ContainerStarted","Data":"98fa6cb7f498a0d59b3eda2ca0e016df071ac07d68817b98f38f2b337fb3bc68"}
Nov 03 16:37:00.283710 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:00.283556    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5" event={"ID":"8ab68de1-044b-4670-8491-493fb1c775e7","Type":"ContainerStarted","Data":"85478a5e89780398d136b009535eac6837e0636c6a5966fa03ed9007c06931dc"}
Nov 03 16:37:00.283710 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:00.283584    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"
Nov 03 16:37:00.309695 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:00.309646    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5" podStartSLOduration=2.309631064 podStartE2EDuration="2.309631064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:00.307429658 +0000 UTC m=+2034.320827808" watchObservedRunningTime="2026-11-03 16:37:00.309631064 +0000 UTC m=+2034.323029209"
Nov 03 16:37:00.498699 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:00.498666    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 16:37:00.498888 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:00.498717    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w" podUID="f9b93c18-e065-4198-897d-2d7dcdcc8d07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 16:37:00.554405 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:00.554319    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:00.828210 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:00.828127    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:01.510372 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:01.510324    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:02.170953 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:02.170920    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:37:02.171408 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:02.171381    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:37:02.172344 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:02.172316    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lmt2w"
Nov 03 16:37:02.636255 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:02.636214    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:03.295820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:03.295756    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:04.132317 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:04.132276    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:04.903527 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:04.903460    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:05.626442 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:05.626389    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:06.344324 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:06.344264    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:07.022703 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:07.022655    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:07.715254 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:07.715215    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:08.444737 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:08.444688    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:09.458641 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:09.458588    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:10.092496 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:10.092464    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:10.704083 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:10.704031    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:11.294028 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:11.294000    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"
Nov 03 16:37:11.372959 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:11.372918    6506 generic.go:356] "Generic (PLEG): container finished" podID="a5b3f102-010a-4db5-a02a-b4b5074dc229" containerID="012476df8f646bc5c116d2ded1d27b59ce51816b25a8c33e86a89b861f8a37a9" exitCode=0
Nov 03 16:37:11.373130 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:11.372980    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp" event={"ID":"a5b3f102-010a-4db5-a02a-b4b5074dc229","Type":"ContainerDied","Data":"012476df8f646bc5c116d2ded1d27b59ce51816b25a8c33e86a89b861f8a37a9"}
Nov 03 16:37:11.373415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:11.373397    6506 scope.go:122] "RemoveContainer" containerID="012476df8f646bc5c116d2ded1d27b59ce51816b25a8c33e86a89b861f8a37a9"
Nov 03 16:37:11.374753 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:11.374720    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:37:11.374852 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:11.374831    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:37:11.374911 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:11.374895    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:37:11.872532 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:11.872484    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:12.030781 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:12.030646    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:12.191616 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:12.191531    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:12.264369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:12.264337    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:37:12.380758 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:12.380725    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp" event={"ID":"a5b3f102-010a-4db5-a02a-b4b5074dc229","Type":"ContainerStarted","Data":"777c78327bd6659b98d0d6acf4ceaabe0590366a084fd4d0278514c6e03a51d2"}
Nov 03 16:37:12.663565 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:12.663529    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:37:12.664608 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:12.664585    6506 patch_prober.go:29] interesting pod/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 16:37:12.664733 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:12.664631    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp" podUID="a5b3f102-010a-4db5-a02a-b4b5074dc229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 16:37:12.784478 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:12.784413    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:12.944317 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:12.944178    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:13.327715 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:13.327627    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:13.531327 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:13.531292    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:13.663401 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:13.663324    6506 patch_prober.go:29] interesting pod/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 16:37:13.663401 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:13.663377    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp" podUID="a5b3f102-010a-4db5-a02a-b4b5074dc229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 16:37:13.761015 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:13.760983    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:14.623689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:14.623648    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:14.663272 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:14.663223    6506 patch_prober.go:29] interesting pod/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 16:37:14.663431 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:14.663314    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp" podUID="a5b3f102-010a-4db5-a02a-b4b5074dc229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 16:37:15.161978 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.161946    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"]
Nov 03 16:37:15.165671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.165649    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.180997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.180972    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"]
Nov 03 16:37:15.186284 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.186260    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-13ba71\"/\"openshift-service-ca.crt\""
Nov 03 16:37:15.186406 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.186328    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-13ba71\"/\"kube-root-ca.crt\""
Nov 03 16:37:15.187412 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.187391    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-13ba71\"/\"default-dockercfg-zj46c\""
Nov 03 16:37:15.187514 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.187471    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-13ba71\"/\"simulator-single-main-13ba71-kserve-self-signed-certs\""
Nov 03 16:37:15.268674 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.268644    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:15.268987 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.268963    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-tmp-dir\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.269075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.269019    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-home\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.269144 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.269084    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4f7\" (UniqueName: \"kubernetes.io/projected/5fc1f101-c669-4a78-b034-123af6f9f9af-kube-api-access-8c4f7\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.269201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.269177    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-dshm\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.269261 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.269198    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-model-cache\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.269261 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.269237    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc1f101-c669-4a78-b034-123af6f9f9af-tls-certs\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.370096 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.370061    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-tmp-dir\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.370287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.370113    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-home\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.370287 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.370244    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4f7\" (UniqueName: \"kubernetes.io/projected/5fc1f101-c669-4a78-b034-123af6f9f9af-kube-api-access-8c4f7\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.370440 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.370330    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-dshm\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.370440 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.370367    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-model-cache\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.370440 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.370417    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc1f101-c669-4a78-b034-123af6f9f9af-tls-certs\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.370598 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.370461    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-home\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.370598 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.370527    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-tmp-dir\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.370776 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.370752    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-model-cache\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.372690 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.372669    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-dshm\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.373010 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.372995    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc1f101-c669-4a78-b034-123af6f9f9af-tls-certs\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.382628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.382578    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4f7\" (UniqueName: \"kubernetes.io/projected/5fc1f101-c669-4a78-b034-123af6f9f9af-kube-api-access-8c4f7\") pod \"simulator-single-main-13ba71-kserve-78689c796c-pjdff\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") " pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.428875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.424818    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:15.477267 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.477237    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:15.663370 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.663330    6506 patch_prober.go:29] interesting pod/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 16:37:15.663875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.663393    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp" podUID="a5b3f102-010a-4db5-a02a-b4b5074dc229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 16:37:15.785192 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:15.785166    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"]
Nov 03 16:37:15.786857 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:37:15.786829    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fc1f101_c669_4a78_b034_123af6f9f9af.slice/crio-7232b90bc42320b086e828a4cf71d79bf473efd1424184277d7229d36f31cb51 WatchSource:0}: Error finding container 7232b90bc42320b086e828a4cf71d79bf473efd1424184277d7229d36f31cb51: Status 404 returned error can't find the container with id 7232b90bc42320b086e828a4cf71d79bf473efd1424184277d7229d36f31cb51
Nov 03 16:37:16.035294 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:16.035192    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:16.403007 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:16.402922    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff" event={"ID":"5fc1f101-c669-4a78-b034-123af6f9f9af","Type":"ContainerStarted","Data":"37eb073058b0194146c1e9d9fb10650bfcad51f3fc24a692369641d8fc4f9bd3"}
Nov 03 16:37:16.403007 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:16.402977    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff" event={"ID":"5fc1f101-c669-4a78-b034-123af6f9f9af","Type":"ContainerStarted","Data":"7232b90bc42320b086e828a4cf71d79bf473efd1424184277d7229d36f31cb51"}
Nov 03 16:37:16.444109 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:16.444057    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff" podStartSLOduration=1.444043001 podStartE2EDuration="1.444043001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:16.44201291 +0000 UTC m=+2050.455411057" watchObservedRunningTime="2026-11-03 16:37:16.444043001 +0000 UTC m=+2050.457441185"
Nov 03 16:37:16.477412 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:16.477385    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:16.663446 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:16.663365    6506 patch_prober.go:29] interesting pod/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 16:37:16.663446 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:16.663422    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp" podUID="a5b3f102-010a-4db5-a02a-b4b5074dc229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 16:37:16.732387 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:16.732347    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:17.593700 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:17.593654    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:17.968473 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:17.967941    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:37:17.969127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:17.969094    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:37:17.972197 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:17.972176    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:37:18.517703 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:18.517656    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:19.240987 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:19.240955    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:19.925943 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:19.925905    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:20.612517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:20.612486    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:21.313617 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:21.313583    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:22.101837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:22.101788    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:22.845752 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:22.845708    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:23.525156 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:23.525124    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:24.276925 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:24.276886    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:25.091579 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:25.091538    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:25.478155 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:25.478069    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:25.491056 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:25.491031    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:25.789736 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:25.789697    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:26.467236 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:26.467208    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:37:26.506537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:26.506492    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:27.214055 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:27.214012    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:28.018631 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:28.018593    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:28.804005 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:37:28.803954    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:37:28.806064 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:28.806035    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:29.510120 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:29.510080    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:30.197994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:30.197961    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:31.020320 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:31.020277    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:31.724710 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:31.724660    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:32.528405 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:32.528335    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:33.430188 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:33.430154    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:34.128330 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:34.128290    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:34.829450 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:34.829415    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:35.633243 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:35.633208    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:36.345337 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:36.345309    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:37.108934 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:37.108899    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:37.808926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:37.808877    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:38.605074 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:38.605040    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:39.313738 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:39.313704    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:40.013923 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:40.013888    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:40.707726 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:40.707691    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:42.681252 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:42.681214    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:42.830349 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:42.830313    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:43.016652 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:43.016562    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:43.795569 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:43.795516    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:44.521979 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:44.521939    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:44.996421 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:44.996341    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"]
Nov 03 16:37:45.000191 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.000170    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"
Nov 03 16:37:45.005369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.005342    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:37:45.005369 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.005345    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:37:45.005502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.005348    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:37:45.005502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.005397    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:37:45.010903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.010882    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"]
Nov 03 16:37:45.166155 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.166127    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d2b6e0-f1fd-463e-bba7-4c05007df112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a2ee2c2-6l7hv\" (UID: \"66d2b6e0-f1fd-463e-bba7-4c05007df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"
Nov 03 16:37:45.166317 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.166176    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ggkv\" (UniqueName: \"kubernetes.io/projected/66d2b6e0-f1fd-463e-bba7-4c05007df112-kube-api-access-5ggkv\") pod \"maas-api-revoke-sub-1a2ee2c2-6l7hv\" (UID: \"66d2b6e0-f1fd-463e-bba7-4c05007df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"
Nov 03 16:37:45.166317 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.166285    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d2b6e0-f1fd-463e-bba7-4c05007df112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a2ee2c2-6l7hv\" (UID: \"66d2b6e0-f1fd-463e-bba7-4c05007df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"
Nov 03 16:37:45.203986 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.203944    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:45.266987 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.266915    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d2b6e0-f1fd-463e-bba7-4c05007df112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a2ee2c2-6l7hv\" (UID: \"66d2b6e0-f1fd-463e-bba7-4c05007df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"
Nov 03 16:37:45.266987 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.266958    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ggkv\" (UniqueName: \"kubernetes.io/projected/66d2b6e0-f1fd-463e-bba7-4c05007df112-kube-api-access-5ggkv\") pod \"maas-api-revoke-sub-1a2ee2c2-6l7hv\" (UID: \"66d2b6e0-f1fd-463e-bba7-4c05007df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"
Nov 03 16:37:45.267222 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.267036    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d2b6e0-f1fd-463e-bba7-4c05007df112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a2ee2c2-6l7hv\" (UID: \"66d2b6e0-f1fd-463e-bba7-4c05007df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"
Nov 03 16:37:45.267857 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.267832    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d2b6e0-f1fd-463e-bba7-4c05007df112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a2ee2c2-6l7hv\" (UID: \"66d2b6e0-f1fd-463e-bba7-4c05007df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"
Nov 03 16:37:45.270012 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.269988    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d2b6e0-f1fd-463e-bba7-4c05007df112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a2ee2c2-6l7hv\" (UID: \"66d2b6e0-f1fd-463e-bba7-4c05007df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"
Nov 03 16:37:45.279645 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.279621    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ggkv\" (UniqueName: \"kubernetes.io/projected/66d2b6e0-f1fd-463e-bba7-4c05007df112-kube-api-access-5ggkv\") pod \"maas-api-revoke-sub-1a2ee2c2-6l7hv\" (UID: \"66d2b6e0-f1fd-463e-bba7-4c05007df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"
Nov 03 16:37:45.314436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.314414    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"
Nov 03 16:37:45.595441 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.595417    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"]
Nov 03 16:37:45.597034 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:37:45.597006    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d2b6e0_f1fd_463e_bba7_4c05007df112.slice/crio-4a2f5a58cf2701c7dce686fdd236e202ba5b01cc99cc6ecf097c8d442055b7b4 WatchSource:0}: Error finding container 4a2f5a58cf2701c7dce686fdd236e202ba5b01cc99cc6ecf097c8d442055b7b4: Status 404 returned error can't find the container with id 4a2f5a58cf2701c7dce686fdd236e202ba5b01cc99cc6ecf097c8d442055b7b4
Nov 03 16:37:45.906384 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:45.906310    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:46.564753 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:46.564712    6506 generic.go:356] "Generic (PLEG): container finished" podID="66d2b6e0-f1fd-463e-bba7-4c05007df112" containerID="852818078bf46ab1c309f4ac2c1940b3592976ce7667c3f4bc914fbd1445f97a" exitCode=0
Nov 03 16:37:46.565264 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:46.564821    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv" event={"ID":"66d2b6e0-f1fd-463e-bba7-4c05007df112","Type":"ContainerDied","Data":"852818078bf46ab1c309f4ac2c1940b3592976ce7667c3f4bc914fbd1445f97a"}
Nov 03 16:37:46.565264 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:46.564878    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv" event={"ID":"66d2b6e0-f1fd-463e-bba7-4c05007df112","Type":"ContainerStarted","Data":"4a2f5a58cf2701c7dce686fdd236e202ba5b01cc99cc6ecf097c8d442055b7b4"}
Nov 03 16:37:46.820973 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:46.820934    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:47.606585 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:47.606547    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:47.720393 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:47.720370    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"
Nov 03 16:37:47.894731 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:47.894659    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d2b6e0-f1fd-463e-bba7-4c05007df112-kube-api-access-5ggkv\" (UniqueName: \"kubernetes.io/projected/66d2b6e0-f1fd-463e-bba7-4c05007df112-kube-api-access-5ggkv\") pod \"66d2b6e0-f1fd-463e-bba7-4c05007df112\" (UID: \"66d2b6e0-f1fd-463e-bba7-4c05007df112\") "
Nov 03 16:37:47.894920 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:47.894740    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d2b6e0-f1fd-463e-bba7-4c05007df112-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d2b6e0-f1fd-463e-bba7-4c05007df112-maas-api-openshift-ca\") pod \"66d2b6e0-f1fd-463e-bba7-4c05007df112\" (UID: \"66d2b6e0-f1fd-463e-bba7-4c05007df112\") "
Nov 03 16:37:47.894920 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:47.894842    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66d2b6e0-f1fd-463e-bba7-4c05007df112-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d2b6e0-f1fd-463e-bba7-4c05007df112-maas-api-xks-ca\") pod \"66d2b6e0-f1fd-463e-bba7-4c05007df112\" (UID: \"66d2b6e0-f1fd-463e-bba7-4c05007df112\") "
Nov 03 16:37:47.895108 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:47.895085    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d2b6e0-f1fd-463e-bba7-4c05007df112-maas-api-openshift-ca" pod "66d2b6e0-f1fd-463e-bba7-4c05007df112" (UID: "66d2b6e0-f1fd-463e-bba7-4c05007df112"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:37:47.895269 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:47.895252    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d2b6e0-f1fd-463e-bba7-4c05007df112-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:37:47.896999 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:47.896979    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d2b6e0-f1fd-463e-bba7-4c05007df112-maas-api-xks-ca" pod "66d2b6e0-f1fd-463e-bba7-4c05007df112" (UID: "66d2b6e0-f1fd-463e-bba7-4c05007df112"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:37:47.897138 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:47.897108    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d2b6e0-f1fd-463e-bba7-4c05007df112-kube-api-access-5ggkv" pod "66d2b6e0-f1fd-463e-bba7-4c05007df112" (UID: "66d2b6e0-f1fd-463e-bba7-4c05007df112"). InnerVolumeSpecName "kube-api-access-5ggkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:37:47.996437 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:47.996407    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ggkv\" (UniqueName: \"kubernetes.io/projected/66d2b6e0-f1fd-463e-bba7-4c05007df112-kube-api-access-5ggkv\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:37:47.996437 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:47.996437    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d2b6e0-f1fd-463e-bba7-4c05007df112-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:37:48.394721 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:48.394684    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:48.576483 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:48.576446    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv" event={"ID":"66d2b6e0-f1fd-463e-bba7-4c05007df112","Type":"ContainerDied","Data":"4a2f5a58cf2701c7dce686fdd236e202ba5b01cc99cc6ecf097c8d442055b7b4"}
Nov 03 16:37:48.576647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:48.576490    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"
Nov 03 16:37:48.576647 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:48.576497    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a2f5a58cf2701c7dce686fdd236e202ba5b01cc99cc6ecf097c8d442055b7b4"
Nov 03 16:37:49.113861 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:49.113813    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:49.817052 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:49.817023    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:50.516276 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:50.516242    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:51.203069 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:51.203037    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:51.899758 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:51.899720    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:52.612768 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:52.612725    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:53.320257 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:53.320225    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:54.036002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:54.035964    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:54.700628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:54.700597    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:55.400101 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:55.400056    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:55.868595 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:55.868559    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:55.972691 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:55.972663    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"]
Nov 03 16:37:55.973377 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:55.973357    6506 cpu_manager.go:414] "containerMap: removing container" podUID="66d2b6e0-f1fd-463e-bba7-4c05007df112" containerName="revoke-keys"
Nov 03 16:37:55.973377 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:55.973378    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d2b6e0-f1fd-463e-bba7-4c05007df112" containerName="revoke-keys"
Nov 03 16:37:55.973581 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:55.973470    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d2b6e0-f1fd-463e-bba7-4c05007df112" containerName="revoke-keys"
Nov 03 16:37:55.976378 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:55.976362    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"
Nov 03 16:37:55.985153 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:55.985131    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:37:55.986417 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:55.986398    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:37:55.986548 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:55.986469    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:37:55.986548 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:55.986511    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:37:55.989748 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:55.989721    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"]
Nov 03 16:37:56.074255 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:56.074223    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f699727e-f7b4-418f-b364-06902a7da9db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb\" (UID: \"f699727e-f7b4-418f-b364-06902a7da9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"
Nov 03 16:37:56.074410 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:56.074336    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f699727e-f7b4-418f-b364-06902a7da9db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb\" (UID: \"f699727e-f7b4-418f-b364-06902a7da9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"
Nov 03 16:37:56.074410 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:56.074373    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtkpx\" (UniqueName: \"kubernetes.io/projected/f699727e-f7b4-418f-b364-06902a7da9db-kube-api-access-dtkpx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb\" (UID: \"f699727e-f7b4-418f-b364-06902a7da9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"
Nov 03 16:37:56.173957 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:56.173873    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:56.175057 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:56.175035    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f699727e-f7b4-418f-b364-06902a7da9db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb\" (UID: \"f699727e-f7b4-418f-b364-06902a7da9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"
Nov 03 16:37:56.175115 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:56.175067    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtkpx\" (UniqueName: \"kubernetes.io/projected/f699727e-f7b4-418f-b364-06902a7da9db-kube-api-access-dtkpx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb\" (UID: \"f699727e-f7b4-418f-b364-06902a7da9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"
Nov 03 16:37:56.175152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:56.175127    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f699727e-f7b4-418f-b364-06902a7da9db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb\" (UID: \"f699727e-f7b4-418f-b364-06902a7da9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"
Nov 03 16:37:56.175842 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:56.175820    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f699727e-f7b4-418f-b364-06902a7da9db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb\" (UID: \"f699727e-f7b4-418f-b364-06902a7da9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"
Nov 03 16:37:56.177528 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:56.177499    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f699727e-f7b4-418f-b364-06902a7da9db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb\" (UID: \"f699727e-f7b4-418f-b364-06902a7da9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"
Nov 03 16:37:56.187354 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:56.187333    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtkpx\" (UniqueName: \"kubernetes.io/projected/f699727e-f7b4-418f-b364-06902a7da9db-kube-api-access-dtkpx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb\" (UID: \"f699727e-f7b4-418f-b364-06902a7da9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"
Nov 03 16:37:56.289731 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:56.289702    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"
Nov 03 16:37:56.568012 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:56.567985    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"]
Nov 03 16:37:56.569298 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:37:56.569271    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf699727e_f7b4_418f_b364_06902a7da9db.slice/crio-58a9f3dd02dd346452d0d9e09e1a6ca3548c95fcc0627e4034ee810408d4dadf WatchSource:0}: Error finding container 58a9f3dd02dd346452d0d9e09e1a6ca3548c95fcc0627e4034ee810408d4dadf: Status 404 returned error can't find the container with id 58a9f3dd02dd346452d0d9e09e1a6ca3548c95fcc0627e4034ee810408d4dadf
Nov 03 16:37:56.622330 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:56.622296    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb" event={"ID":"f699727e-f7b4-418f-b364-06902a7da9db","Type":"ContainerStarted","Data":"58a9f3dd02dd346452d0d9e09e1a6ca3548c95fcc0627e4034ee810408d4dadf"}
Nov 03 16:37:57.121414 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:57.121318    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:57.628646 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:57.628611    6506 generic.go:356] "Generic (PLEG): container finished" podID="f699727e-f7b4-418f-b364-06902a7da9db" containerID="d7a2d72f8b34e80e33f00faebf90997a36ac2969a91bc9767f42f0cc86ed4afc" exitCode=0
Nov 03 16:37:57.628872 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:57.628723    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb" event={"ID":"f699727e-f7b4-418f-b364-06902a7da9db","Type":"ContainerDied","Data":"d7a2d72f8b34e80e33f00faebf90997a36ac2969a91bc9767f42f0cc86ed4afc"}
Nov 03 16:37:57.919363 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:57.919323    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:58.643682 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:58.643652    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:58.776035 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:58.776012    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"
Nov 03 16:37:58.795540 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:37:58.795508    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:37:58.811200 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:58.811177    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f699727e-f7b4-418f-b364-06902a7da9db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f699727e-f7b4-418f-b364-06902a7da9db-maas-api-openshift-ca\") pod \"f699727e-f7b4-418f-b364-06902a7da9db\" (UID: \"f699727e-f7b4-418f-b364-06902a7da9db\") "
Nov 03 16:37:58.811334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:58.811258    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f699727e-f7b4-418f-b364-06902a7da9db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f699727e-f7b4-418f-b364-06902a7da9db-maas-api-xks-ca\") pod \"f699727e-f7b4-418f-b364-06902a7da9db\" (UID: \"f699727e-f7b4-418f-b364-06902a7da9db\") "
Nov 03 16:37:58.811334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:58.811328    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f699727e-f7b4-418f-b364-06902a7da9db-kube-api-access-dtkpx\" (UniqueName: \"kubernetes.io/projected/f699727e-f7b4-418f-b364-06902a7da9db-kube-api-access-dtkpx\") pod \"f699727e-f7b4-418f-b364-06902a7da9db\" (UID: \"f699727e-f7b4-418f-b364-06902a7da9db\") "
Nov 03 16:37:58.811594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:58.811572    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f699727e-f7b4-418f-b364-06902a7da9db-maas-api-openshift-ca" pod "f699727e-f7b4-418f-b364-06902a7da9db" (UID: "f699727e-f7b4-418f-b364-06902a7da9db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:37:58.811687 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:58.811673    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f699727e-f7b4-418f-b364-06902a7da9db-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:37:58.813624 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:58.813537    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f699727e-f7b4-418f-b364-06902a7da9db-maas-api-xks-ca" pod "f699727e-f7b4-418f-b364-06902a7da9db" (UID: "f699727e-f7b4-418f-b364-06902a7da9db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:37:58.813942 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:58.813760    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f699727e-f7b4-418f-b364-06902a7da9db-kube-api-access-dtkpx" pod "f699727e-f7b4-418f-b364-06902a7da9db" (UID: "f699727e-f7b4-418f-b364-06902a7da9db"). InnerVolumeSpecName "kube-api-access-dtkpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:37:58.913173 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:58.913090    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtkpx\" (UniqueName: \"kubernetes.io/projected/f699727e-f7b4-418f-b364-06902a7da9db-kube-api-access-dtkpx\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:37:58.913173 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:58.913129    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f699727e-f7b4-418f-b364-06902a7da9db-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:37:59.321610 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:59.321579    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:37:59.641720 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:59.641632    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb" event={"ID":"f699727e-f7b4-418f-b364-06902a7da9db","Type":"ContainerDied","Data":"58a9f3dd02dd346452d0d9e09e1a6ca3548c95fcc0627e4034ee810408d4dadf"}
Nov 03 16:37:59.641720 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:59.641674    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"
Nov 03 16:37:59.641967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:37:59.641678    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58a9f3dd02dd346452d0d9e09e1a6ca3548c95fcc0627e4034ee810408d4dadf"
Nov 03 16:38:00.034892 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:00.034860    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:00.774876 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:00.774840    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"]
Nov 03 16:38:00.902834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:00.902775    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:01.091817 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:01.091701    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:01.660542 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:01.660511    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:02.405308 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:02.405278    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:03.116580 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:03.116544    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:05.203568 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:05.203528    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:05.927367 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:05.927332    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:06.122937 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:06.122907    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"]
Nov 03 16:38:06.238703 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:06.238625    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:06.407997 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:06.407964    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:06.564460 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:06.564420    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:06.604339 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:06.604303    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"]
Nov 03 16:38:06.604867 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:06.604842    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-13ba71-lc2mb"]
Nov 03 16:38:06.719689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:06.719651    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:07.186219 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:07.186164    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:08.102641 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:08.102612    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f699727e-f7b4-418f-b364-06902a7da9db" path="/var/lib/kubelet/pods/f699727e-f7b4-418f-b364-06902a7da9db/volumes"
Nov 03 16:38:10.640359 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:10.640326    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:11.124172 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:11.124121    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5" podUID="8ab68de1-044b-4670-8491-493fb1c775e7" containerName="payload-pre-processing" containerID="cri-o://98fa6cb7f498a0d59b3eda2ca0e016df071ac07d68817b98f38f2b337fb3bc68" gracePeriod=25
Nov 03 16:38:11.292175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:11.292139    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5" podUID="8ab68de1-044b-4670-8491-493fb1c775e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:38:13.321715 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:13.321684    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:13.481056 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:13.481027    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:13.594232 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:13.594143    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"]
Nov 03 16:38:13.594470 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:13.594439    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp" podUID="a5b3f102-010a-4db5-a02a-b4b5074dc229" containerName="istio-proxy" containerID="cri-o://777c78327bd6659b98d0d6acf4ceaabe0590366a084fd4d0278514c6e03a51d2" gracePeriod=30
Nov 03 16:38:13.690415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:13.690377    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:15.795383 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:15.795333    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff" podUID="5fc1f101-c669-4a78-b034-123af6f9f9af" containerName="main" containerID="cri-o://37eb073058b0194146c1e9d9fb10650bfcad51f3fc24a692369641d8fc4f9bd3" gracePeriod=45
Nov 03 16:38:16.044657 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.044634    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:38:16.195931 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.195850    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-model-cache\") pod \"5fc1f101-c669-4a78-b034-123af6f9f9af\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") "
Nov 03 16:38:16.195931 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.195920    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-dshm\") pod \"5fc1f101-c669-4a78-b034-123af6f9f9af\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") "
Nov 03 16:38:16.196175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.195969    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc1f101-c669-4a78-b034-123af6f9f9af-kube-api-access-8c4f7\" (UniqueName: \"kubernetes.io/projected/5fc1f101-c669-4a78-b034-123af6f9f9af-kube-api-access-8c4f7\") pod \"5fc1f101-c669-4a78-b034-123af6f9f9af\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") "
Nov 03 16:38:16.196175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.196014    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-home\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-home\") pod \"5fc1f101-c669-4a78-b034-123af6f9f9af\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") "
Nov 03 16:38:16.196175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.196079    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-tmp-dir\") pod \"5fc1f101-c669-4a78-b034-123af6f9f9af\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") "
Nov 03 16:38:16.196175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.196074    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-model-cache" pod "5fc1f101-c669-4a78-b034-123af6f9f9af" (UID: "5fc1f101-c669-4a78-b034-123af6f9f9af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:38:16.196175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.196128    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fc1f101-c669-4a78-b034-123af6f9f9af-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc1f101-c669-4a78-b034-123af6f9f9af-tls-certs\") pod \"5fc1f101-c669-4a78-b034-123af6f9f9af\" (UID: \"5fc1f101-c669-4a78-b034-123af6f9f9af\") "
Nov 03 16:38:16.196411 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.196252    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-home" pod "5fc1f101-c669-4a78-b034-123af6f9f9af" (UID: "5fc1f101-c669-4a78-b034-123af6f9f9af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:38:16.196411 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.196299    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-tmp-dir" pod "5fc1f101-c669-4a78-b034-123af6f9f9af" (UID: "5fc1f101-c669-4a78-b034-123af6f9f9af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:38:16.196671 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.196649    6506 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-home\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:16.196750 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.196724    6506 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-tmp-dir\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:16.196750 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.196739    6506 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-model-cache\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:16.198544 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.198519    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc1f101-c669-4a78-b034-123af6f9f9af-kube-api-access-8c4f7" pod "5fc1f101-c669-4a78-b034-123af6f9f9af" (UID: "5fc1f101-c669-4a78-b034-123af6f9f9af"). InnerVolumeSpecName "kube-api-access-8c4f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:38:16.198628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.198542    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-dshm" pod "5fc1f101-c669-4a78-b034-123af6f9f9af" (UID: "5fc1f101-c669-4a78-b034-123af6f9f9af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:38:16.198628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.198598    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fc1f101-c669-4a78-b034-123af6f9f9af-tls-certs" pod "5fc1f101-c669-4a78-b034-123af6f9f9af" (UID: "5fc1f101-c669-4a78-b034-123af6f9f9af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:38:16.298263 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.298200    6506 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc1f101-c669-4a78-b034-123af6f9f9af-tls-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:16.298263 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.298230    6506 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc1f101-c669-4a78-b034-123af6f9f9af-dshm\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:16.298263 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.298242    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c4f7\" (UniqueName: \"kubernetes.io/projected/5fc1f101-c669-4a78-b034-123af6f9f9af-kube-api-access-8c4f7\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:16.740743 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.740706    6506 generic.go:356] "Generic (PLEG): container finished" podID="5fc1f101-c669-4a78-b034-123af6f9f9af" containerID="37eb073058b0194146c1e9d9fb10650bfcad51f3fc24a692369641d8fc4f9bd3" exitCode=0
Nov 03 16:38:16.740941 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.740750    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"
Nov 03 16:38:16.740941 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.740813    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff" event={"ID":"5fc1f101-c669-4a78-b034-123af6f9f9af","Type":"ContainerDied","Data":"37eb073058b0194146c1e9d9fb10650bfcad51f3fc24a692369641d8fc4f9bd3"}
Nov 03 16:38:16.740941 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.740860    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff" event={"ID":"5fc1f101-c669-4a78-b034-123af6f9f9af","Type":"ContainerDied","Data":"7232b90bc42320b086e828a4cf71d79bf473efd1424184277d7229d36f31cb51"}
Nov 03 16:38:16.740941 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.740877    6506 scope.go:122] "RemoveContainer" containerID="37eb073058b0194146c1e9d9fb10650bfcad51f3fc24a692369641d8fc4f9bd3"
Nov 03 16:38:16.750652 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.750636    6506 scope.go:122] "RemoveContainer" containerID="37eb073058b0194146c1e9d9fb10650bfcad51f3fc24a692369641d8fc4f9bd3"
Nov 03 16:38:16.750912 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:38:16.750895    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37eb073058b0194146c1e9d9fb10650bfcad51f3fc24a692369641d8fc4f9bd3\": container with ID starting with 37eb073058b0194146c1e9d9fb10650bfcad51f3fc24a692369641d8fc4f9bd3 not found: ID does not exist" containerID="37eb073058b0194146c1e9d9fb10650bfcad51f3fc24a692369641d8fc4f9bd3"
Nov 03 16:38:16.750973 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.750920    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37eb073058b0194146c1e9d9fb10650bfcad51f3fc24a692369641d8fc4f9bd3"} err="failed to get container status \"37eb073058b0194146c1e9d9fb10650bfcad51f3fc24a692369641d8fc4f9bd3\": rpc error: code = NotFound desc = could not find container \"37eb073058b0194146c1e9d9fb10650bfcad51f3fc24a692369641d8fc4f9bd3\": container with ID starting with 37eb073058b0194146c1e9d9fb10650bfcad51f3fc24a692369641d8fc4f9bd3 not found: ID does not exist"
Nov 03 16:38:16.763396 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.763376    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"]
Nov 03 16:38:16.767002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:16.766980    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-13ba71/simulator-single-main-13ba71-kserve-78689c796c-pjdff"]
Nov 03 16:38:17.869118 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:17.869080    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"]
Nov 03 16:38:17.870588 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:17.870556    6506 cpu_manager.go:414] "containerMap: removing container" podUID="f699727e-f7b4-418f-b364-06902a7da9db" containerName="revoke-keys"
Nov 03 16:38:17.870751 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:17.870737    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f699727e-f7b4-418f-b364-06902a7da9db" containerName="revoke-keys"
Nov 03 16:38:17.870948 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:17.870935    6506 cpu_manager.go:414] "containerMap: removing container" podUID="5fc1f101-c669-4a78-b034-123af6f9f9af" containerName="main"
Nov 03 16:38:17.871047 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:17.871033    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc1f101-c669-4a78-b034-123af6f9f9af" containerName="main"
Nov 03 16:38:17.871327 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:17.871311    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc1f101-c669-4a78-b034-123af6f9f9af" containerName="main"
Nov 03 16:38:17.871436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:17.871425    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="f699727e-f7b4-418f-b364-06902a7da9db" containerName="revoke-keys"
Nov 03 16:38:17.876449 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:17.876426    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:17.888490 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:17.888468    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c44cdadd-kserve-self-signed-certs\""
Nov 03 16:38:17.890018 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:17.889992    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"]
Nov 03 16:38:17.957858 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:17.957821    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:17.970583 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:17.970554    6506 patch_prober.go:29] interesting pod/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:38:17.970752 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:17.970611    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp" podUID="a5b3f102-010a-4db5-a02a-b4b5074dc229" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:38:18.016242 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.016202    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-tmp-dir\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.016242 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.016246    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-model-cache\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.016517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.016290    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-dshm\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.016517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.016464    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4zjx\" (UniqueName: \"kubernetes.io/projected/b58fa98d-6a30-4a64-a009-c4d4e94139e0-kube-api-access-m4zjx\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.016517 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.016505    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58fa98d-6a30-4a64-a009-c4d4e94139e0-tls-certs\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.016679 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.016524    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-home\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.102288 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.102256    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc1f101-c669-4a78-b034-123af6f9f9af" path="/var/lib/kubelet/pods/5fc1f101-c669-4a78-b034-123af6f9f9af/volumes"
Nov 03 16:38:18.117249 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.117222    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4zjx\" (UniqueName: \"kubernetes.io/projected/b58fa98d-6a30-4a64-a009-c4d4e94139e0-kube-api-access-m4zjx\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.117434 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.117274    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58fa98d-6a30-4a64-a009-c4d4e94139e0-tls-certs\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.117434 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.117301    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-home\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.117434 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.117338    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-tmp-dir\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.117593 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.117464    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-model-cache\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.117659 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.117590    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-dshm\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.117748 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.117728    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-home\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.117829 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.117811    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-tmp-dir\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.117881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.117828    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-model-cache\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.119887 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.119824    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-dshm\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.120099 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.120079    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58fa98d-6a30-4a64-a009-c4d4e94139e0-tls-certs\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.129833 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.129810    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4zjx\" (UniqueName: \"kubernetes.io/projected/b58fa98d-6a30-4a64-a009-c4d4e94139e0-kube-api-access-m4zjx\") pod \"e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") " pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.188818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.188753    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:18.469992 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.469960    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"]
Nov 03 16:38:18.472172 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:38:18.472141    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb58fa98d_6a30_4a64_a009_c4d4e94139e0.slice/crio-8b3b7984ac911220791d286cfaac8ffcc216b828f3cf4a48d7971895d8c1a495 WatchSource:0}: Error finding container 8b3b7984ac911220791d286cfaac8ffcc216b828f3cf4a48d7971895d8c1a495: Status 404 returned error can't find the container with id 8b3b7984ac911220791d286cfaac8ffcc216b828f3cf4a48d7971895d8c1a495
Nov 03 16:38:18.755754 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.755667    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf" event={"ID":"b58fa98d-6a30-4a64-a009-c4d4e94139e0","Type":"ContainerStarted","Data":"5ebf8a7c52ce9498333c7918635131395e4a9afd2a4b321d3daad01fa717678f"}
Nov 03 16:38:18.755754 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.755716    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf" event={"ID":"b58fa98d-6a30-4a64-a009-c4d4e94139e0","Type":"ContainerStarted","Data":"8b3b7984ac911220791d286cfaac8ffcc216b828f3cf4a48d7971895d8c1a495"}
Nov 03 16:38:18.757811 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.757772    6506 generic.go:356] "Generic (PLEG): container finished" podID="a5b3f102-010a-4db5-a02a-b4b5074dc229" containerID="777c78327bd6659b98d0d6acf4ceaabe0590366a084fd4d0278514c6e03a51d2" exitCode=0
Nov 03 16:38:18.757926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.757839    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp" event={"ID":"a5b3f102-010a-4db5-a02a-b4b5074dc229","Type":"ContainerDied","Data":"777c78327bd6659b98d0d6acf4ceaabe0590366a084fd4d0278514c6e03a51d2"}
Nov 03 16:38:18.757926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.757890    6506 scope.go:122] "RemoveContainer" containerID="012476df8f646bc5c116d2ded1d27b59ce51816b25a8c33e86a89b861f8a37a9"
Nov 03 16:38:18.779968 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.779905    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf" podStartSLOduration=1.779885611 podStartE2EDuration="1.779885611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:18.776071418 +0000 UTC m=+2112.789469568" watchObservedRunningTime="2026-11-03 16:38:18.779885611 +0000 UTC m=+2112.793283758"
Nov 03 16:38:18.841027 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.841003    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:38:18.923675 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.923648    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-workload-certs\") pod \"a5b3f102-010a-4db5-a02a-b4b5074dc229\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") "
Nov 03 16:38:18.924168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.923701    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b3f102-010a-4db5-a02a-b4b5074dc229-kube-api-access-rm7mg\" (UniqueName: \"kubernetes.io/projected/a5b3f102-010a-4db5-a02a-b4b5074dc229-kube-api-access-rm7mg\") pod \"a5b3f102-010a-4db5-a02a-b4b5074dc229\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") "
Nov 03 16:38:18.924168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.923730    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-token\" (UniqueName: \"kubernetes.io/projected/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-token\") pod \"a5b3f102-010a-4db5-a02a-b4b5074dc229\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") "
Nov 03 16:38:18.924168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.923763    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5b3f102-010a-4db5-a02a-b4b5074dc229-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b3f102-010a-4db5-a02a-b4b5074dc229-istiod-ca-cert\") pod \"a5b3f102-010a-4db5-a02a-b4b5074dc229\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") "
Nov 03 16:38:18.924168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.923820    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-workload-socket\") pod \"a5b3f102-010a-4db5-a02a-b4b5074dc229\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") "
Nov 03 16:38:18.924168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.923964    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-data\") pod \"a5b3f102-010a-4db5-a02a-b4b5074dc229\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") "
Nov 03 16:38:18.924168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.923992    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-workload-certs" pod "a5b3f102-010a-4db5-a02a-b4b5074dc229" (UID: "a5b3f102-010a-4db5-a02a-b4b5074dc229"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:38:18.924168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.924097    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-podinfo\") pod \"a5b3f102-010a-4db5-a02a-b4b5074dc229\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") "
Nov 03 16:38:18.924168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.924115    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-workload-socket" pod "a5b3f102-010a-4db5-a02a-b4b5074dc229" (UID: "a5b3f102-010a-4db5-a02a-b4b5074dc229"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:38:18.924168 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.924158    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-envoy\") pod \"a5b3f102-010a-4db5-a02a-b4b5074dc229\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") "
Nov 03 16:38:18.924637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.924220    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-credential-socket\") pod \"a5b3f102-010a-4db5-a02a-b4b5074dc229\" (UID: \"a5b3f102-010a-4db5-a02a-b4b5074dc229\") "
Nov 03 16:38:18.924637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.924229    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-data" pod "a5b3f102-010a-4db5-a02a-b4b5074dc229" (UID: "a5b3f102-010a-4db5-a02a-b4b5074dc229"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:38:18.924637 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.924282    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b3f102-010a-4db5-a02a-b4b5074dc229-istiod-ca-cert" pod "a5b3f102-010a-4db5-a02a-b4b5074dc229" (UID: "a5b3f102-010a-4db5-a02a-b4b5074dc229"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:38:18.924874 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.924844    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:18.924967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.924872    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b3f102-010a-4db5-a02a-b4b5074dc229-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:18.924967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.924891    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:18.924967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.924905    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:18.925442 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.925417    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-credential-socket" pod "a5b3f102-010a-4db5-a02a-b4b5074dc229" (UID: "a5b3f102-010a-4db5-a02a-b4b5074dc229"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:38:18.926275 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.926255    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-token" pod "a5b3f102-010a-4db5-a02a-b4b5074dc229" (UID: "a5b3f102-010a-4db5-a02a-b4b5074dc229"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:38:18.926352 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.926320    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b3f102-010a-4db5-a02a-b4b5074dc229-kube-api-access-rm7mg" pod "a5b3f102-010a-4db5-a02a-b4b5074dc229" (UID: "a5b3f102-010a-4db5-a02a-b4b5074dc229"). InnerVolumeSpecName "kube-api-access-rm7mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:38:18.926727 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.926710    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-podinfo" pod "a5b3f102-010a-4db5-a02a-b4b5074dc229" (UID: "a5b3f102-010a-4db5-a02a-b4b5074dc229"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:38:18.926867 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:18.926853    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-envoy" pod "a5b3f102-010a-4db5-a02a-b4b5074dc229" (UID: "a5b3f102-010a-4db5-a02a-b4b5074dc229"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:38:19.026348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:19.026312    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:19.026348 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:19.026346    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b3f102-010a-4db5-a02a-b4b5074dc229-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:19.026547 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:19.026358    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm7mg\" (UniqueName: \"kubernetes.io/projected/a5b3f102-010a-4db5-a02a-b4b5074dc229-kube-api-access-rm7mg\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:19.026547 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:19.026368    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:19.026547 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:19.026376    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b3f102-010a-4db5-a02a-b4b5074dc229-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:19.189621 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:19.189582    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:19.765032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:19.764996    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"
Nov 03 16:38:19.765032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:19.765016    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp" event={"ID":"a5b3f102-010a-4db5-a02a-b4b5074dc229","Type":"ContainerDied","Data":"4082e60d0c43faef9430778cceb3cb649b33867f29baaa701b9d4010270cc515"}
Nov 03 16:38:19.765300 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:19.765080    6506 scope.go:122] "RemoveContainer" containerID="777c78327bd6659b98d0d6acf4ceaabe0590366a084fd4d0278514c6e03a51d2"
Nov 03 16:38:19.816984 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:38:19.816943    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b3f102_010a_4db5_a02a_b4b5074dc229.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b3f102_010a_4db5_a02a_b4b5074dc229.slice/crio-4082e60d0c43faef9430778cceb3cb649b33867f29baaa701b9d4010270cc515\": RecentStats: unable to find data in memory cache]"
Nov 03 16:38:19.833782 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:19.833750    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"]
Nov 03 16:38:19.847480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:19.847447    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-13ba71-openshift-default-56d5dbc997p84cp"]
Nov 03 16:38:20.106478 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:20.106430    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5b3f102-010a-4db5-a02a-b4b5074dc229" path="/var/lib/kubelet/pods/a5b3f102-010a-4db5-a02a-b4b5074dc229/volumes"
Nov 03 16:38:21.306856 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:21.305884    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5" podUID="8ab68de1-044b-4670-8491-493fb1c775e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:38:28.189582 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:28.189546    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:28.202029 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:28.202005    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:28.796340 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:38:28.796297    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:38:28.852480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:28.852450    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:38:31.292131 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:38:31.292099    6506 logging.go:55] [core] [Channel #737 SubChannel #738]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.131:50052", ServerName: "10.133.0.131:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.131:50052: connect: connection refused"
Nov 03 16:38:31.379310 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.379284    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"
Nov 03 16:38:31.407524 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.407461    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab68de1-044b-4670-8491-493fb1c775e7-kube-api-access-2d4nf\" (UniqueName: \"kubernetes.io/projected/8ab68de1-044b-4670-8491-493fb1c775e7-kube-api-access-2d4nf\") pod \"8ab68de1-044b-4670-8491-493fb1c775e7\" (UID: \"8ab68de1-044b-4670-8491-493fb1c775e7\") "
Nov 03 16:38:31.407524 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.407494    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab68de1-044b-4670-8491-493fb1c775e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab68de1-044b-4670-8491-493fb1c775e7-plugins-config-volume\") pod \"8ab68de1-044b-4670-8491-493fb1c775e7\" (UID: \"8ab68de1-044b-4670-8491-493fb1c775e7\") "
Nov 03 16:38:31.407927 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.407908    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab68de1-044b-4670-8491-493fb1c775e7-plugins-config-volume" pod "8ab68de1-044b-4670-8491-493fb1c775e7" (UID: "8ab68de1-044b-4670-8491-493fb1c775e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:38:31.409917 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.409893    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab68de1-044b-4670-8491-493fb1c775e7-kube-api-access-2d4nf" pod "8ab68de1-044b-4670-8491-493fb1c775e7" (UID: "8ab68de1-044b-4670-8491-493fb1c775e7"). InnerVolumeSpecName "kube-api-access-2d4nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:38:31.508286 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.508258    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d4nf\" (UniqueName: \"kubernetes.io/projected/8ab68de1-044b-4670-8491-493fb1c775e7-kube-api-access-2d4nf\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:31.508286 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.508282    6506 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab68de1-044b-4670-8491-493fb1c775e7-plugins-config-volume\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:31.855009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.854978    6506 generic.go:356] "Generic (PLEG): container finished" podID="8ab68de1-044b-4670-8491-493fb1c775e7" containerID="98fa6cb7f498a0d59b3eda2ca0e016df071ac07d68817b98f38f2b337fb3bc68" exitCode=0
Nov 03 16:38:31.855009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.855023    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"
Nov 03 16:38:31.855228 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.855045    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5" event={"ID":"8ab68de1-044b-4670-8491-493fb1c775e7","Type":"ContainerDied","Data":"98fa6cb7f498a0d59b3eda2ca0e016df071ac07d68817b98f38f2b337fb3bc68"}
Nov 03 16:38:31.855228 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.855097    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5" event={"ID":"8ab68de1-044b-4670-8491-493fb1c775e7","Type":"ContainerDied","Data":"85478a5e89780398d136b009535eac6837e0636c6a5966fa03ed9007c06931dc"}
Nov 03 16:38:31.855228 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.855125    6506 scope.go:122] "RemoveContainer" containerID="98fa6cb7f498a0d59b3eda2ca0e016df071ac07d68817b98f38f2b337fb3bc68"
Nov 03 16:38:31.864960 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.864938    6506 scope.go:122] "RemoveContainer" containerID="98fa6cb7f498a0d59b3eda2ca0e016df071ac07d68817b98f38f2b337fb3bc68"
Nov 03 16:38:31.865261 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:38:31.865241    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98fa6cb7f498a0d59b3eda2ca0e016df071ac07d68817b98f38f2b337fb3bc68\": container with ID starting with 98fa6cb7f498a0d59b3eda2ca0e016df071ac07d68817b98f38f2b337fb3bc68 not found: ID does not exist" containerID="98fa6cb7f498a0d59b3eda2ca0e016df071ac07d68817b98f38f2b337fb3bc68"
Nov 03 16:38:31.865326 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.865272    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98fa6cb7f498a0d59b3eda2ca0e016df071ac07d68817b98f38f2b337fb3bc68"} err="failed to get container status \"98fa6cb7f498a0d59b3eda2ca0e016df071ac07d68817b98f38f2b337fb3bc68\": rpc error: code = NotFound desc = could not find container \"98fa6cb7f498a0d59b3eda2ca0e016df071ac07d68817b98f38f2b337fb3bc68\": container with ID starting with 98fa6cb7f498a0d59b3eda2ca0e016df071ac07d68817b98f38f2b337fb3bc68 not found: ID does not exist"
Nov 03 16:38:31.877848 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.877819    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"]
Nov 03 16:38:31.882007 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:31.881981    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5"]
Nov 03 16:38:32.102279 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:32.102244    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab68de1-044b-4670-8491-493fb1c775e7" path="/var/lib/kubelet/pods/8ab68de1-044b-4670-8491-493fb1c775e7/volumes"
Nov 03 16:38:32.292534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:32.292489    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-13ba71-85c5b4fgl5" podUID="8ab68de1-044b-4670-8491-493fb1c775e7" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.131:50052\" within 1s: context deadline exceeded"
Nov 03 16:38:34.282652 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.282618    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"]
Nov 03 16:38:34.283094 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.283085    6506 cpu_manager.go:414] "containerMap: removing container" podUID="a5b3f102-010a-4db5-a02a-b4b5074dc229" containerName="istio-proxy"
Nov 03 16:38:34.283138 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.283098    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b3f102-010a-4db5-a02a-b4b5074dc229" containerName="istio-proxy"
Nov 03 16:38:34.283138 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.283130    6506 cpu_manager.go:414] "containerMap: removing container" podUID="8ab68de1-044b-4670-8491-493fb1c775e7" containerName="payload-pre-processing"
Nov 03 16:38:34.283138 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.283135    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab68de1-044b-4670-8491-493fb1c775e7" containerName="payload-pre-processing"
Nov 03 16:38:34.283274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.283210    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b3f102-010a-4db5-a02a-b4b5074dc229" containerName="istio-proxy"
Nov 03 16:38:34.283274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.283223    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab68de1-044b-4670-8491-493fb1c775e7" containerName="payload-pre-processing"
Nov 03 16:38:34.287608 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.287588    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.297430 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.297402    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c44cdadd-kserve-self-signed-certs\""
Nov 03 16:38:34.298980 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.298957    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"]
Nov 03 16:38:34.334488 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.334456    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3448f2c-2758-48fb-bc52-907bf7af1789-tls-certs\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.334620 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.334497    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-dshm\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.334620 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.334525    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-model-cache\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.334707 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.334682    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-tmp-dir\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.334763 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.334749    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-home\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.334838 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.334789    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c8cs\" (UniqueName: \"kubernetes.io/projected/f3448f2c-2758-48fb-bc52-907bf7af1789-kube-api-access-4c8cs\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.389746 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.389722    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:34.436194 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.436158    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3448f2c-2758-48fb-bc52-907bf7af1789-tls-certs\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.436381 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.436206    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-dshm\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.436468 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.436391    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-model-cache\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.436531 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.436468    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-tmp-dir\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.436590 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.436537    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-home\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.436642 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.436583    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c8cs\" (UniqueName: \"kubernetes.io/projected/f3448f2c-2758-48fb-bc52-907bf7af1789-kube-api-access-4c8cs\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.436980 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.436923    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-tmp-dir\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.437197 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.437172    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-home\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.437415 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.437395    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-model-cache\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.438935 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.438916    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-dshm\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.439089 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.439073    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3448f2c-2758-48fb-bc52-907bf7af1789-tls-certs\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.449203 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.449183    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c8cs\" (UniqueName: \"kubernetes.io/projected/f3448f2c-2758-48fb-bc52-907bf7af1789-kube-api-access-4c8cs\") pod \"e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") " pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.601296 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.601206    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:34.882331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:34.882305    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"]
Nov 03 16:38:34.884934 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:38:34.884904    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3448f2c_2758_48fb_bc52_907bf7af1789.slice/crio-a3ef05acb3bef0ed9a9ea2dac11cfe8d568ac953b0ab057975f5f615b4ab2375 WatchSource:0}: Error finding container a3ef05acb3bef0ed9a9ea2dac11cfe8d568ac953b0ab057975f5f615b4ab2375: Status 404 returned error can't find the container with id a3ef05acb3bef0ed9a9ea2dac11cfe8d568ac953b0ab057975f5f615b4ab2375
Nov 03 16:38:35.878964 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:35.878931    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n" event={"ID":"f3448f2c-2758-48fb-bc52-907bf7af1789","Type":"ContainerStarted","Data":"56a0a2d4ffdb468f4f5660f60f2b8b0c192cdec1cc572ff84e1c46b8637f6aac"}
Nov 03 16:38:35.879372 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:35.878971    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n" event={"ID":"f3448f2c-2758-48fb-bc52-907bf7af1789","Type":"ContainerStarted","Data":"a3ef05acb3bef0ed9a9ea2dac11cfe8d568ac953b0ab057975f5f615b4ab2375"}
Nov 03 16:38:35.901007 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:35.900949    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n" podStartSLOduration=1.90093495 podStartE2EDuration="1.90093495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:35.899351934 +0000 UTC m=+2129.912750082" watchObservedRunningTime="2026-11-03 16:38:35.90093495 +0000 UTC m=+2129.914333133"
Nov 03 16:38:36.602045 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:36.602005    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:37.415645 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:37.415591    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:38.466278 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:38.466239    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:42.353201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:42.353161    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:42.508855 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:42.508825    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:44.193016 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:44.192987    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:38:44.601500 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:44.601452    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:44.614112 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:44.614087    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:45.006513 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:45.006429    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:38:51.450032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.449998    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"]
Nov 03 16:38:51.450497 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.450367    6506 cpu_manager.go:414] "containerMap: removing container" podUID="a5b3f102-010a-4db5-a02a-b4b5074dc229" containerName="istio-proxy"
Nov 03 16:38:51.450497 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.450377    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b3f102-010a-4db5-a02a-b4b5074dc229" containerName="istio-proxy"
Nov 03 16:38:51.450497 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.450471    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b3f102-010a-4db5-a02a-b4b5074dc229" containerName="istio-proxy"
Nov 03 16:38:51.453638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.453621    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"
Nov 03 16:38:51.457915 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.457888    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:38:51.458026 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.457892    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:38:51.459233 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.459219    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:38:51.459592 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.459560    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:38:51.463852 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.463833    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"]
Nov 03 16:38:51.608539 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.608504    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0e04549-54bd-4add-bca3-5b849bfc1669-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-62b2q\" (UID: \"f0e04549-54bd-4add-bca3-5b849bfc1669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"
Nov 03 16:38:51.608762 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.608737    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0e04549-54bd-4add-bca3-5b849bfc1669-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-62b2q\" (UID: \"f0e04549-54bd-4add-bca3-5b849bfc1669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"
Nov 03 16:38:51.608865 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.608779    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v47t4\" (UniqueName: \"kubernetes.io/projected/f0e04549-54bd-4add-bca3-5b849bfc1669-kube-api-access-v47t4\") pod \"maas-api-revoke-sub-452b6d7b-62b2q\" (UID: \"f0e04549-54bd-4add-bca3-5b849bfc1669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"
Nov 03 16:38:51.709879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.709773    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0e04549-54bd-4add-bca3-5b849bfc1669-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-62b2q\" (UID: \"f0e04549-54bd-4add-bca3-5b849bfc1669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"
Nov 03 16:38:51.709879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.709839    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47t4\" (UniqueName: \"kubernetes.io/projected/f0e04549-54bd-4add-bca3-5b849bfc1669-kube-api-access-v47t4\") pod \"maas-api-revoke-sub-452b6d7b-62b2q\" (UID: \"f0e04549-54bd-4add-bca3-5b849bfc1669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"
Nov 03 16:38:51.709879 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.709873    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0e04549-54bd-4add-bca3-5b849bfc1669-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-62b2q\" (UID: \"f0e04549-54bd-4add-bca3-5b849bfc1669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"
Nov 03 16:38:51.710558 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.710533    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0e04549-54bd-4add-bca3-5b849bfc1669-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-62b2q\" (UID: \"f0e04549-54bd-4add-bca3-5b849bfc1669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"
Nov 03 16:38:51.712279 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.712257    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0e04549-54bd-4add-bca3-5b849bfc1669-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-62b2q\" (UID: \"f0e04549-54bd-4add-bca3-5b849bfc1669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"
Nov 03 16:38:51.722950 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.722926    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v47t4\" (UniqueName: \"kubernetes.io/projected/f0e04549-54bd-4add-bca3-5b849bfc1669-kube-api-access-v47t4\") pod \"maas-api-revoke-sub-452b6d7b-62b2q\" (UID: \"f0e04549-54bd-4add-bca3-5b849bfc1669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"
Nov 03 16:38:51.771592 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:51.771566    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"
Nov 03 16:38:52.053831 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:52.053774    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"]
Nov 03 16:38:52.055422 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:38:52.055391    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e04549_54bd_4add_bca3_5b849bfc1669.slice/crio-badfcea8d618aba13753e11b542c7c61685a6cbb0fca02fb9c346e59abe3d888 WatchSource:0}: Error finding container badfcea8d618aba13753e11b542c7c61685a6cbb0fca02fb9c346e59abe3d888: Status 404 returned error can't find the container with id badfcea8d618aba13753e11b542c7c61685a6cbb0fca02fb9c346e59abe3d888
Nov 03 16:38:53.044174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:53.044138    6506 generic.go:356] "Generic (PLEG): container finished" podID="f0e04549-54bd-4add-bca3-5b849bfc1669" containerID="5ed8e2a7eff8f6e39000864dd899a54b901f3c6dfe207e82818c5e4555f7b357" exitCode=0
Nov 03 16:38:53.044626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:53.044219    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q" event={"ID":"f0e04549-54bd-4add-bca3-5b849bfc1669","Type":"ContainerDied","Data":"5ed8e2a7eff8f6e39000864dd899a54b901f3c6dfe207e82818c5e4555f7b357"}
Nov 03 16:38:53.044626 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:53.044282    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q" event={"ID":"f0e04549-54bd-4add-bca3-5b849bfc1669","Type":"ContainerStarted","Data":"badfcea8d618aba13753e11b542c7c61685a6cbb0fca02fb9c346e59abe3d888"}
Nov 03 16:38:54.840175 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:54.840144    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"
Nov 03 16:38:55.044454 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:55.044412    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0e04549-54bd-4add-bca3-5b849bfc1669-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0e04549-54bd-4add-bca3-5b849bfc1669-maas-api-xks-ca\") pod \"f0e04549-54bd-4add-bca3-5b849bfc1669\" (UID: \"f0e04549-54bd-4add-bca3-5b849bfc1669\") "
Nov 03 16:38:55.050935 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:55.050904    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e04549-54bd-4add-bca3-5b849bfc1669-kube-api-access-v47t4\" (UniqueName: \"kubernetes.io/projected/f0e04549-54bd-4add-bca3-5b849bfc1669-kube-api-access-v47t4\") pod \"f0e04549-54bd-4add-bca3-5b849bfc1669\" (UID: \"f0e04549-54bd-4add-bca3-5b849bfc1669\") "
Nov 03 16:38:55.051162 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:55.051141    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0e04549-54bd-4add-bca3-5b849bfc1669-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0e04549-54bd-4add-bca3-5b849bfc1669-maas-api-openshift-ca\") pod \"f0e04549-54bd-4add-bca3-5b849bfc1669\" (UID: \"f0e04549-54bd-4add-bca3-5b849bfc1669\") "
Nov 03 16:38:55.064292 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:55.064249    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e04549-54bd-4add-bca3-5b849bfc1669-maas-api-openshift-ca" pod "f0e04549-54bd-4add-bca3-5b849bfc1669" (UID: "f0e04549-54bd-4add-bca3-5b849bfc1669"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:38:55.093882 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:55.090925    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e04549-54bd-4add-bca3-5b849bfc1669-kube-api-access-v47t4" pod "f0e04549-54bd-4add-bca3-5b849bfc1669" (UID: "f0e04549-54bd-4add-bca3-5b849bfc1669"). InnerVolumeSpecName "kube-api-access-v47t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:38:55.093882 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:55.091386    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0e04549-54bd-4add-bca3-5b849bfc1669-maas-api-xks-ca" pod "f0e04549-54bd-4add-bca3-5b849bfc1669" (UID: "f0e04549-54bd-4add-bca3-5b849bfc1669"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:38:55.139850 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:55.131287    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q" event={"ID":"f0e04549-54bd-4add-bca3-5b849bfc1669","Type":"ContainerDied","Data":"badfcea8d618aba13753e11b542c7c61685a6cbb0fca02fb9c346e59abe3d888"}
Nov 03 16:38:55.139850 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:55.131350    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="badfcea8d618aba13753e11b542c7c61685a6cbb0fca02fb9c346e59abe3d888"
Nov 03 16:38:55.139850 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:55.131423    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"
Nov 03 16:38:55.172820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:55.169336    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0e04549-54bd-4add-bca3-5b849bfc1669-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:55.173072 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:55.173048    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v47t4\" (UniqueName: \"kubernetes.io/projected/f0e04549-54bd-4add-bca3-5b849bfc1669-kube-api-access-v47t4\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:55.173209 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:38:55.173193    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0e04549-54bd-4add-bca3-5b849bfc1669-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:38:58.796131 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:38:58.796091    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:39:02.347642 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:02.347602    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"]
Nov 03 16:39:02.508456 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:02.508418    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:04.785333 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:04.785296    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"]
Nov 03 16:39:04.900668 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:04.900628    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:07.073821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:07.072559    6506 scope.go:122] "RemoveContainer" containerID="5d3306cc68eb533b98375dfc1a5098681764716558b3260c8fe6822e6fb8ffa2"
Nov 03 16:39:07.956075 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:07.956032    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"]
Nov 03 16:39:07.956486 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:07.956469    6506 cpu_manager.go:414] "containerMap: removing container" podUID="f0e04549-54bd-4add-bca3-5b849bfc1669" containerName="revoke-keys"
Nov 03 16:39:07.956591 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:07.956488    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e04549-54bd-4add-bca3-5b849bfc1669" containerName="revoke-keys"
Nov 03 16:39:07.956651 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:07.956598    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e04549-54bd-4add-bca3-5b849bfc1669" containerName="revoke-keys"
Nov 03 16:39:07.968672 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:07.968634    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"
Nov 03 16:39:07.971751 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:07.971723    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"]
Nov 03 16:39:07.972887 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:07.972674    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:39:07.974837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:07.973928    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:39:07.974837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:07.974089    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:39:07.974837 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:07.974170    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:39:08.065503 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:08.065476    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-x2twg\" (UID: \"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"
Nov 03 16:39:08.065736 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:08.065711    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mthgv\" (UniqueName: \"kubernetes.io/projected/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-kube-api-access-mthgv\") pod \"maas-api-revoke-sub-3e31a506-x2twg\" (UID: \"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"
Nov 03 16:39:08.065807 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:08.065778    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-x2twg\" (UID: \"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"
Nov 03 16:39:08.166669 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:08.166474    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mthgv\" (UniqueName: \"kubernetes.io/projected/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-kube-api-access-mthgv\") pod \"maas-api-revoke-sub-3e31a506-x2twg\" (UID: \"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"
Nov 03 16:39:08.166669 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:08.166560    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-x2twg\" (UID: \"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"
Nov 03 16:39:08.166669 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:08.166612    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-x2twg\" (UID: \"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"
Nov 03 16:39:08.167688 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:08.167659    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-x2twg\" (UID: \"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"
Nov 03 16:39:08.169987 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:08.169958    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-x2twg\" (UID: \"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"
Nov 03 16:39:08.180227 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:08.180185    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mthgv\" (UniqueName: \"kubernetes.io/projected/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-kube-api-access-mthgv\") pod \"maas-api-revoke-sub-3e31a506-x2twg\" (UID: \"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"
Nov 03 16:39:08.285678 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:08.285642    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"
Nov 03 16:39:08.598261 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:08.598231    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"]
Nov 03 16:39:08.600688 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:39:08.600631    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01ce01d_b2f9_45d0_b8aa_a3777b34aa65.slice/crio-8f76093efeb0763c87a1f19818ba4ab33a990ed3e75448f2040b3d1b920d6353 WatchSource:0}: Error finding container 8f76093efeb0763c87a1f19818ba4ab33a990ed3e75448f2040b3d1b920d6353: Status 404 returned error can't find the container with id 8f76093efeb0763c87a1f19818ba4ab33a990ed3e75448f2040b3d1b920d6353
Nov 03 16:39:09.278820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:09.277909    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg" event={"ID":"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65","Type":"ContainerStarted","Data":"791f9b2799f8720fecd2c3a7699db13cccf7c3be28129f6d438520045538f2ac"}
Nov 03 16:39:09.278820 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:09.277978    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg" event={"ID":"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65","Type":"ContainerStarted","Data":"8f76093efeb0763c87a1f19818ba4ab33a990ed3e75448f2040b3d1b920d6353"}
Nov 03 16:39:10.354774 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:10.354025    6506 generic.go:356] "Generic (PLEG): container finished" podID="f01ce01d-b2f9-45d0-b8aa-a3777b34aa65" containerID="791f9b2799f8720fecd2c3a7699db13cccf7c3be28129f6d438520045538f2ac" exitCode=0
Nov 03 16:39:10.354774 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:10.354240    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg" event={"ID":"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65","Type":"ContainerDied","Data":"791f9b2799f8720fecd2c3a7699db13cccf7c3be28129f6d438520045538f2ac"}
Nov 03 16:39:10.799808 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:10.799754    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:11.491565 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:11.491540    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"
Nov 03 16:39:11.614174 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:11.614094    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-kube-api-access-mthgv\" (UniqueName: \"kubernetes.io/projected/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-kube-api-access-mthgv\") pod \"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65\" (UID: \"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65\") "
Nov 03 16:39:11.614326 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:11.614179    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-maas-api-xks-ca\") pod \"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65\" (UID: \"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65\") "
Nov 03 16:39:11.614326 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:11.614248    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-maas-api-openshift-ca\") pod \"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65\" (UID: \"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65\") "
Nov 03 16:39:11.614620 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:11.614599    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-maas-api-openshift-ca" pod "f01ce01d-b2f9-45d0-b8aa-a3777b34aa65" (UID: "f01ce01d-b2f9-45d0-b8aa-a3777b34aa65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:39:11.616299 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:11.616277    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-kube-api-access-mthgv" pod "f01ce01d-b2f9-45d0-b8aa-a3777b34aa65" (UID: "f01ce01d-b2f9-45d0-b8aa-a3777b34aa65"). InnerVolumeSpecName "kube-api-access-mthgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:39:11.616408 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:11.616300    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-maas-api-xks-ca" pod "f01ce01d-b2f9-45d0-b8aa-a3777b34aa65" (UID: "f01ce01d-b2f9-45d0-b8aa-a3777b34aa65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:39:11.715780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:11.715751    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mthgv\" (UniqueName: \"kubernetes.io/projected/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-kube-api-access-mthgv\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:11.715780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:11.715777    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:11.715996 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:11.715787    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:12.366630 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:12.366548    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"
Nov 03 16:39:12.366630 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:12.366560    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg" event={"ID":"f01ce01d-b2f9-45d0-b8aa-a3777b34aa65","Type":"ContainerDied","Data":"8f76093efeb0763c87a1f19818ba4ab33a990ed3e75448f2040b3d1b920d6353"}
Nov 03 16:39:12.366630 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:12.366604    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f76093efeb0763c87a1f19818ba4ab33a990ed3e75448f2040b3d1b920d6353"
Nov 03 16:39:17.369210 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.369109    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf" podUID="b58fa98d-6a30-4a64-a009-c4d4e94139e0" containerName="main" containerID="cri-o://5ebf8a7c52ce9498333c7918635131395e4a9afd2a4b321d3daad01fa717678f" gracePeriod=45
Nov 03 16:39:17.637927 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.637902    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:39:17.770440 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.770408    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-home\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-home\") pod \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") "
Nov 03 16:39:17.770440 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.770441    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b58fa98d-6a30-4a64-a009-c4d4e94139e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/b58fa98d-6a30-4a64-a009-c4d4e94139e0-tls-certs\") pod \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") "
Nov 03 16:39:17.770638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.770482    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-model-cache\") pod \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") "
Nov 03 16:39:17.770638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.770510    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-dshm\") pod \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") "
Nov 03 16:39:17.770638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.770544    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b58fa98d-6a30-4a64-a009-c4d4e94139e0-kube-api-access-m4zjx\" (UniqueName: \"kubernetes.io/projected/b58fa98d-6a30-4a64-a009-c4d4e94139e0-kube-api-access-m4zjx\") pod \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") "
Nov 03 16:39:17.770638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.770596    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-tmp-dir\") pod \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\" (UID: \"b58fa98d-6a30-4a64-a009-c4d4e94139e0\") "
Nov 03 16:39:17.770894 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.770715    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-home" pod "b58fa98d-6a30-4a64-a009-c4d4e94139e0" (UID: "b58fa98d-6a30-4a64-a009-c4d4e94139e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:17.770894 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.770786    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-model-cache" pod "b58fa98d-6a30-4a64-a009-c4d4e94139e0" (UID: "b58fa98d-6a30-4a64-a009-c4d4e94139e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:17.771017 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.770930    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-tmp-dir" pod "b58fa98d-6a30-4a64-a009-c4d4e94139e0" (UID: "b58fa98d-6a30-4a64-a009-c4d4e94139e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:17.771017 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.770949    6506 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-model-cache\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:17.771017 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.770970    6506 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-home\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:17.773031 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.772999    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b58fa98d-6a30-4a64-a009-c4d4e94139e0-tls-certs" pod "b58fa98d-6a30-4a64-a009-c4d4e94139e0" (UID: "b58fa98d-6a30-4a64-a009-c4d4e94139e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:39:17.773136 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.773042    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-dshm" pod "b58fa98d-6a30-4a64-a009-c4d4e94139e0" (UID: "b58fa98d-6a30-4a64-a009-c4d4e94139e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:17.773136 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.773049    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58fa98d-6a30-4a64-a009-c4d4e94139e0-kube-api-access-m4zjx" pod "b58fa98d-6a30-4a64-a009-c4d4e94139e0" (UID: "b58fa98d-6a30-4a64-a009-c4d4e94139e0"). InnerVolumeSpecName "kube-api-access-m4zjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:39:17.871731 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.871661    6506 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58fa98d-6a30-4a64-a009-c4d4e94139e0-tls-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:17.871731 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.871692    6506 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-dshm\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:17.871731 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.871704    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4zjx\" (UniqueName: \"kubernetes.io/projected/b58fa98d-6a30-4a64-a009-c4d4e94139e0-kube-api-access-m4zjx\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:17.871731 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:17.871732    6506 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58fa98d-6a30-4a64-a009-c4d4e94139e0-tmp-dir\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:18.399555 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:18.399520    6506 generic.go:356] "Generic (PLEG): container finished" podID="b58fa98d-6a30-4a64-a009-c4d4e94139e0" containerID="5ebf8a7c52ce9498333c7918635131395e4a9afd2a4b321d3daad01fa717678f" exitCode=0
Nov 03 16:39:18.400021 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:18.399571    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"
Nov 03 16:39:18.400021 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:18.399586    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf" event={"ID":"b58fa98d-6a30-4a64-a009-c4d4e94139e0","Type":"ContainerDied","Data":"5ebf8a7c52ce9498333c7918635131395e4a9afd2a4b321d3daad01fa717678f"}
Nov 03 16:39:18.400021 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:18.399625    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf" event={"ID":"b58fa98d-6a30-4a64-a009-c4d4e94139e0","Type":"ContainerDied","Data":"8b3b7984ac911220791d286cfaac8ffcc216b828f3cf4a48d7971895d8c1a495"}
Nov 03 16:39:18.400021 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:18.399647    6506 scope.go:122] "RemoveContainer" containerID="5ebf8a7c52ce9498333c7918635131395e4a9afd2a4b321d3daad01fa717678f"
Nov 03 16:39:18.408816 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:18.408780    6506 scope.go:122] "RemoveContainer" containerID="5ebf8a7c52ce9498333c7918635131395e4a9afd2a4b321d3daad01fa717678f"
Nov 03 16:39:18.409089 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:39:18.409066    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ebf8a7c52ce9498333c7918635131395e4a9afd2a4b321d3daad01fa717678f\": container with ID starting with 5ebf8a7c52ce9498333c7918635131395e4a9afd2a4b321d3daad01fa717678f not found: ID does not exist" containerID="5ebf8a7c52ce9498333c7918635131395e4a9afd2a4b321d3daad01fa717678f"
Nov 03 16:39:18.409219 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:18.409098    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebf8a7c52ce9498333c7918635131395e4a9afd2a4b321d3daad01fa717678f"} err="failed to get container status \"5ebf8a7c52ce9498333c7918635131395e4a9afd2a4b321d3daad01fa717678f\": rpc error: code = NotFound desc = could not find container \"5ebf8a7c52ce9498333c7918635131395e4a9afd2a4b321d3daad01fa717678f\": container with ID starting with 5ebf8a7c52ce9498333c7918635131395e4a9afd2a4b321d3daad01fa717678f not found: ID does not exist"
Nov 03 16:39:18.424296 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:18.424267    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"]
Nov 03 16:39:18.429705 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:18.429685    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c44cdadd-kserve-67559bd4d5-wd5mf"]
Nov 03 16:39:19.841821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:19.840959    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n" podUID="f3448f2c-2758-48fb-bc52-907bf7af1789" containerName="main" containerID="cri-o://56a0a2d4ffdb468f4f5660f60f2b8b0c192cdec1cc572ff84e1c46b8637f6aac" gracePeriod=45
Nov 03 16:39:19.999252 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:19.998821    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n" podUID="f3448f2c-2758-48fb-bc52-907bf7af1789" containerName="main" probeResult="failure" output="Get \"https://10.133.0.136:8000/ready\": dial tcp 10.133.0.136:8000: connect: connection refused"
Nov 03 16:39:20.003123 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:39:20.003089    6506 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3448f2c_2758_48fb_bc52_907bf7af1789.slice/crio-56a0a2d4ffdb468f4f5660f60f2b8b0c192cdec1cc572ff84e1c46b8637f6aac.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:39:20.103480 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.103399    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b58fa98d-6a30-4a64-a009-c4d4e94139e0" path="/var/lib/kubelet/pods/b58fa98d-6a30-4a64-a009-c4d4e94139e0/volumes"
Nov 03 16:39:20.509486 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.509169    6506 generic.go:356] "Generic (PLEG): container finished" podID="f3448f2c-2758-48fb-bc52-907bf7af1789" containerID="56a0a2d4ffdb468f4f5660f60f2b8b0c192cdec1cc572ff84e1c46b8637f6aac" exitCode=0
Nov 03 16:39:20.509486 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.509353    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n" event={"ID":"f3448f2c-2758-48fb-bc52-907bf7af1789","Type":"ContainerDied","Data":"56a0a2d4ffdb468f4f5660f60f2b8b0c192cdec1cc572ff84e1c46b8637f6aac"}
Nov 03 16:39:20.509486 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.509390    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n" event={"ID":"f3448f2c-2758-48fb-bc52-907bf7af1789","Type":"ContainerDied","Data":"a3ef05acb3bef0ed9a9ea2dac11cfe8d568ac953b0ab057975f5f615b4ab2375"}
Nov 03 16:39:20.509486 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.509409    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3ef05acb3bef0ed9a9ea2dac11cfe8d568ac953b0ab057975f5f615b4ab2375"
Nov 03 16:39:20.522484 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.522213    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:39:20.626868 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.615855    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-dshm\") pod \"f3448f2c-2758-48fb-bc52-907bf7af1789\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") "
Nov 03 16:39:20.646830 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.627123    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-model-cache\") pod \"f3448f2c-2758-48fb-bc52-907bf7af1789\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") "
Nov 03 16:39:20.647764 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.647741    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-home\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-home\") pod \"f3448f2c-2758-48fb-bc52-907bf7af1789\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") "
Nov 03 16:39:20.655110 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.647622    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-model-cache" pod "f3448f2c-2758-48fb-bc52-907bf7af1789" (UID: "f3448f2c-2758-48fb-bc52-907bf7af1789"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:20.668876 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.660014    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-tmp-dir" pod "f3448f2c-2758-48fb-bc52-907bf7af1789" (UID: "f3448f2c-2758-48fb-bc52-907bf7af1789"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:20.669087 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.660074    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-home" pod "f3448f2c-2758-48fb-bc52-907bf7af1789" (UID: "f3448f2c-2758-48fb-bc52-907bf7af1789"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:20.669240 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.654983    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-tmp-dir\") pod \"f3448f2c-2758-48fb-bc52-907bf7af1789\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") "
Nov 03 16:39:20.669504 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.669483    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3448f2c-2758-48fb-bc52-907bf7af1789-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3448f2c-2758-48fb-bc52-907bf7af1789-tls-certs\") pod \"f3448f2c-2758-48fb-bc52-907bf7af1789\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") "
Nov 03 16:39:20.683493 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.683418    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3448f2c-2758-48fb-bc52-907bf7af1789-kube-api-access-4c8cs\" (UniqueName: \"kubernetes.io/projected/f3448f2c-2758-48fb-bc52-907bf7af1789-kube-api-access-4c8cs\") pod \"f3448f2c-2758-48fb-bc52-907bf7af1789\" (UID: \"f3448f2c-2758-48fb-bc52-907bf7af1789\") "
Nov 03 16:39:20.740862 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.740830    6506 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-home\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:20.741182 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.741157    6506 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-tmp-dir\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:20.741360 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.741346    6506 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-model-cache\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:20.741492 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.687008    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-dshm" pod "f3448f2c-2758-48fb-bc52-907bf7af1789" (UID: "f3448f2c-2758-48fb-bc52-907bf7af1789"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:20.741616 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.734941    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3448f2c-2758-48fb-bc52-907bf7af1789-tls-certs" pod "f3448f2c-2758-48fb-bc52-907bf7af1789" (UID: "f3448f2c-2758-48fb-bc52-907bf7af1789"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:39:20.748987 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.748939    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3448f2c-2758-48fb-bc52-907bf7af1789-kube-api-access-4c8cs" pod "f3448f2c-2758-48fb-bc52-907bf7af1789" (UID: "f3448f2c-2758-48fb-bc52-907bf7af1789"). InnerVolumeSpecName "kube-api-access-4c8cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:39:20.845829 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.842393    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c8cs\" (UniqueName: \"kubernetes.io/projected/f3448f2c-2758-48fb-bc52-907bf7af1789-kube-api-access-4c8cs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:20.845829 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.842437    6506 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3448f2c-2758-48fb-bc52-907bf7af1789-dshm\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:20.845829 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:20.842453    6506 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3448f2c-2758-48fb-bc52-907bf7af1789-tls-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:21.516925 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:21.516878    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"
Nov 03 16:39:21.546239 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:21.546198    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"]
Nov 03 16:39:21.553821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:21.551120    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c44cdadd-kserve-56948dffd5-6ds2n"]
Nov 03 16:39:22.105894 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:22.105856    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3448f2c-2758-48fb-bc52-907bf7af1789" path="/var/lib/kubelet/pods/f3448f2c-2758-48fb-bc52-907bf7af1789/volumes"
Nov 03 16:39:28.424289 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.424252    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"]
Nov 03 16:39:28.424847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.424661    6506 cpu_manager.go:414] "containerMap: removing container" podUID="f3448f2c-2758-48fb-bc52-907bf7af1789" containerName="main"
Nov 03 16:39:28.424847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.424674    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3448f2c-2758-48fb-bc52-907bf7af1789" containerName="main"
Nov 03 16:39:28.424847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.424698    6506 cpu_manager.go:414] "containerMap: removing container" podUID="b58fa98d-6a30-4a64-a009-c4d4e94139e0" containerName="main"
Nov 03 16:39:28.424847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.424703    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b58fa98d-6a30-4a64-a009-c4d4e94139e0" containerName="main"
Nov 03 16:39:28.424847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.424710    6506 cpu_manager.go:414] "containerMap: removing container" podUID="f01ce01d-b2f9-45d0-b8aa-a3777b34aa65" containerName="revoke-keys"
Nov 03 16:39:28.424847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.424715    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01ce01d-b2f9-45d0-b8aa-a3777b34aa65" containerName="revoke-keys"
Nov 03 16:39:28.424847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.424780    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="b58fa98d-6a30-4a64-a009-c4d4e94139e0" containerName="main"
Nov 03 16:39:28.424847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.424789    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01ce01d-b2f9-45d0-b8aa-a3777b34aa65" containerName="revoke-keys"
Nov 03 16:39:28.424847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.424819    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3448f2c-2758-48fb-bc52-907bf7af1789" containerName="main"
Nov 03 16:39:28.429164 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.429148    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"
Nov 03 16:39:28.437281 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.437249    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:39:28.437452 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.437295    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:39:28.437452 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.437313    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:39:28.437452 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.437333    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:39:28.441516 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.441457    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"]
Nov 03 16:39:28.546239 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.546205    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb773898-2zdw7\" (UID: \"0f3c3d72-f12c-4d43-80ca-8170dc053d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"
Nov 03 16:39:28.546395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.546270    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb773898-2zdw7\" (UID: \"0f3c3d72-f12c-4d43-80ca-8170dc053d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"
Nov 03 16:39:28.546488 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.546386    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c6qr\" (UniqueName: \"kubernetes.io/projected/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-kube-api-access-6c6qr\") pod \"maas-api-revoke-sub-cb773898-2zdw7\" (UID: \"0f3c3d72-f12c-4d43-80ca-8170dc053d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"
Nov 03 16:39:28.647842 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.647773    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb773898-2zdw7\" (UID: \"0f3c3d72-f12c-4d43-80ca-8170dc053d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"
Nov 03 16:39:28.648002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.647863    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c6qr\" (UniqueName: \"kubernetes.io/projected/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-kube-api-access-6c6qr\") pod \"maas-api-revoke-sub-cb773898-2zdw7\" (UID: \"0f3c3d72-f12c-4d43-80ca-8170dc053d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"
Nov 03 16:39:28.648002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.647966    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb773898-2zdw7\" (UID: \"0f3c3d72-f12c-4d43-80ca-8170dc053d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"
Nov 03 16:39:28.648609 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.648591    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb773898-2zdw7\" (UID: \"0f3c3d72-f12c-4d43-80ca-8170dc053d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"
Nov 03 16:39:28.650260 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.650241    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb773898-2zdw7\" (UID: \"0f3c3d72-f12c-4d43-80ca-8170dc053d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"
Nov 03 16:39:28.660756 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.660731    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c6qr\" (UniqueName: \"kubernetes.io/projected/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-kube-api-access-6c6qr\") pod \"maas-api-revoke-sub-cb773898-2zdw7\" (UID: \"0f3c3d72-f12c-4d43-80ca-8170dc053d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"
Nov 03 16:39:28.743833 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:28.743731    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"
Nov 03 16:39:28.807533 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:39:28.807499    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:39:29.023464 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:29.023438    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"]
Nov 03 16:39:29.025039 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:39:29.025007    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3c3d72_f12c_4d43_80ca_8170dc053d1b.slice/crio-9570f76588b0f79bdd5821040813f37fcfd8595c9095dce6359d8aaa05ba08bc WatchSource:0}: Error finding container 9570f76588b0f79bdd5821040813f37fcfd8595c9095dce6359d8aaa05ba08bc: Status 404 returned error can't find the container with id 9570f76588b0f79bdd5821040813f37fcfd8595c9095dce6359d8aaa05ba08bc
Nov 03 16:39:29.595612 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:29.595535    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7" event={"ID":"0f3c3d72-f12c-4d43-80ca-8170dc053d1b","Type":"ContainerStarted","Data":"340cf7480c2c00ed41445dff63251ffb8019ad2307c804a3013d3326da89ff01"}
Nov 03 16:39:29.595612 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:29.595583    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7" event={"ID":"0f3c3d72-f12c-4d43-80ca-8170dc053d1b","Type":"ContainerStarted","Data":"9570f76588b0f79bdd5821040813f37fcfd8595c9095dce6359d8aaa05ba08bc"}
Nov 03 16:39:29.617334 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:29.617277    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7" podStartSLOduration=1.617260519 podStartE2EDuration="1.617260519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:29.616248821 +0000 UTC m=+2183.629646966" watchObservedRunningTime="2026-11-03 16:39:29.617260519 +0000 UTC m=+2183.630658665"
Nov 03 16:39:30.601894 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:30.601857    6506 generic.go:356] "Generic (PLEG): container finished" podID="0f3c3d72-f12c-4d43-80ca-8170dc053d1b" containerID="340cf7480c2c00ed41445dff63251ffb8019ad2307c804a3013d3326da89ff01" exitCode=0
Nov 03 16:39:30.602282 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:30.601939    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7" event={"ID":"0f3c3d72-f12c-4d43-80ca-8170dc053d1b","Type":"ContainerDied","Data":"340cf7480c2c00ed41445dff63251ffb8019ad2307c804a3013d3326da89ff01"}
Nov 03 16:39:31.741996 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:31.741971    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"
Nov 03 16:39:31.779505 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:31.779481    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-maas-api-openshift-ca\") pod \"0f3c3d72-f12c-4d43-80ca-8170dc053d1b\" (UID: \"0f3c3d72-f12c-4d43-80ca-8170dc053d1b\") "
Nov 03 16:39:31.779638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:31.779526    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-maas-api-xks-ca\") pod \"0f3c3d72-f12c-4d43-80ca-8170dc053d1b\" (UID: \"0f3c3d72-f12c-4d43-80ca-8170dc053d1b\") "
Nov 03 16:39:31.779638 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:31.779624    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-kube-api-access-6c6qr\" (UniqueName: \"kubernetes.io/projected/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-kube-api-access-6c6qr\") pod \"0f3c3d72-f12c-4d43-80ca-8170dc053d1b\" (UID: \"0f3c3d72-f12c-4d43-80ca-8170dc053d1b\") "
Nov 03 16:39:31.779886 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:31.779864    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-maas-api-openshift-ca" pod "0f3c3d72-f12c-4d43-80ca-8170dc053d1b" (UID: "0f3c3d72-f12c-4d43-80ca-8170dc053d1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:39:31.781750 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:31.781725    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-maas-api-xks-ca" pod "0f3c3d72-f12c-4d43-80ca-8170dc053d1b" (UID: "0f3c3d72-f12c-4d43-80ca-8170dc053d1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:39:31.781895 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:31.781809    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-kube-api-access-6c6qr" pod "0f3c3d72-f12c-4d43-80ca-8170dc053d1b" (UID: "0f3c3d72-f12c-4d43-80ca-8170dc053d1b"). InnerVolumeSpecName "kube-api-access-6c6qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:39:31.880915 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:31.880846    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-maas-api-openshift-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:31.880915 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:31.880873    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-maas-api-xks-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:31.880915 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:31.880883    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c6qr\" (UniqueName: \"kubernetes.io/projected/0f3c3d72-f12c-4d43-80ca-8170dc053d1b-kube-api-access-6c6qr\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:32.614995 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:32.614964    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7" event={"ID":"0f3c3d72-f12c-4d43-80ca-8170dc053d1b","Type":"ContainerDied","Data":"9570f76588b0f79bdd5821040813f37fcfd8595c9095dce6359d8aaa05ba08bc"}
Nov 03 16:39:32.615170 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:32.615015    6506 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9570f76588b0f79bdd5821040813f37fcfd8595c9095dce6359d8aaa05ba08bc"
Nov 03 16:39:32.615170 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:32.614976    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"
Nov 03 16:39:40.810354 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:40.810319    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"]
Nov 03 16:39:40.810732 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:40.810709    6506 cpu_manager.go:414] "containerMap: removing container" podUID="0f3c3d72-f12c-4d43-80ca-8170dc053d1b" containerName="revoke-keys"
Nov 03 16:39:40.810732 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:40.810720    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3c3d72-f12c-4d43-80ca-8170dc053d1b" containerName="revoke-keys"
Nov 03 16:39:40.810823 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:40.810785    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3c3d72-f12c-4d43-80ca-8170dc053d1b" containerName="revoke-keys"
Nov 03 16:39:40.813570 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:40.813554    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:40.859724 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:40.859702    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 16:39:40.859724 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:40.859702    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 16:39:40.859926 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:40.859701    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v65q5\""
Nov 03 16:39:40.864260 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:40.864238    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"]
Nov 03 16:39:40.869293 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:40.869270    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:39:40.872258 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:40.872231    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:39:40.966333 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:40.966298    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8687a740-c507-4497-ae4a-60bb890a4070-metrics-certs\") pod \"maas-api-87dc4db6-n5pvl\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:40.966525 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:40.966361    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8687a740-c507-4497-ae4a-60bb890a4070-openshift-service-ca\") pod \"maas-api-87dc4db6-n5pvl\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:40.966584 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:40.966559    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8687a740-c507-4497-ae4a-60bb890a4070-maas-api-tls\") pod \"maas-api-87dc4db6-n5pvl\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:40.966625 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:40.966596    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf9b4\" (UniqueName: \"kubernetes.io/projected/8687a740-c507-4497-ae4a-60bb890a4070-kube-api-access-bf9b4\") pod \"maas-api-87dc4db6-n5pvl\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:41.067834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:41.067740    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8687a740-c507-4497-ae4a-60bb890a4070-metrics-certs\") pod \"maas-api-87dc4db6-n5pvl\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:41.067834 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:41.067783    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8687a740-c507-4497-ae4a-60bb890a4070-openshift-service-ca\") pod \"maas-api-87dc4db6-n5pvl\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:41.068056 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:41.067872    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8687a740-c507-4497-ae4a-60bb890a4070-maas-api-tls\") pod \"maas-api-87dc4db6-n5pvl\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:41.068056 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:41.067897    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf9b4\" (UniqueName: \"kubernetes.io/projected/8687a740-c507-4497-ae4a-60bb890a4070-kube-api-access-bf9b4\") pod \"maas-api-87dc4db6-n5pvl\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:41.068534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:41.068511    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8687a740-c507-4497-ae4a-60bb890a4070-openshift-service-ca\") pod \"maas-api-87dc4db6-n5pvl\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:41.070297 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:41.070277    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8687a740-c507-4497-ae4a-60bb890a4070-metrics-certs\") pod \"maas-api-87dc4db6-n5pvl\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:41.070397 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:41.070353    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8687a740-c507-4497-ae4a-60bb890a4070-maas-api-tls\") pod \"maas-api-87dc4db6-n5pvl\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:41.082546 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:41.082523    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf9b4\" (UniqueName: \"kubernetes.io/projected/8687a740-c507-4497-ae4a-60bb890a4070-kube-api-access-bf9b4\") pod \"maas-api-87dc4db6-n5pvl\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:41.126530 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:41.126500    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:41.402932 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:41.402905    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"]
Nov 03 16:39:41.405360 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:39:41.405334    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8687a740_c507_4497_ae4a_60bb890a4070.slice/crio-5bd5231ffd02664ce003af0a7f8fc8a5e7818e70ec460033557faaa685161bd4 WatchSource:0}: Error finding container 5bd5231ffd02664ce003af0a7f8fc8a5e7818e70ec460033557faaa685161bd4: Status 404 returned error can't find the container with id 5bd5231ffd02664ce003af0a7f8fc8a5e7818e70ec460033557faaa685161bd4
Nov 03 16:39:41.406734 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:41.406716    6506 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:39:41.661297 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:41.661217    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl" event={"ID":"8687a740-c507-4497-ae4a-60bb890a4070","Type":"ContainerStarted","Data":"5bd5231ffd02664ce003af0a7f8fc8a5e7818e70ec460033557faaa685161bd4"}
Nov 03 16:39:42.668237 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:42.668199    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl" event={"ID":"8687a740-c507-4497-ae4a-60bb890a4070","Type":"ContainerStarted","Data":"1550a67faf45b34373874992a9bb473f2fc6d1391a0c2540b774484a06282ae4"}
Nov 03 16:39:42.668650 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:42.668334    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:42.692670 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:42.692618    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl" podStartSLOduration=2.295447723 podStartE2EDuration="2.692605678s" totalImagesPullingTime="397.157955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:42.689833505 +0000 UTC m=+2196.703231660" watchObservedRunningTime="2026-11-03 16:39:42.692605678 +0000 UTC m=+2196.706003824"
Nov 03 16:39:43.030558 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:43.030525    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"]
Nov 03 16:39:43.033639 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:43.033615    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nt4jl"]
Nov 03 16:39:44.101847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:44.101814    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136e618c-25fb-4f4d-bdcd-13af4f832c34" path="/var/lib/kubelet/pods/136e618c-25fb-4f4d-bdcd-13af4f832c34/volumes"
Nov 03 16:39:47.849904 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:47.849870    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:48.246448 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.246372    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"]
Nov 03 16:39:48.250734 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.250710    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.258569 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.258549    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-56d83be3-openshift-default-dockercfg-6swkc\""
Nov 03 16:39:48.267531 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.267496    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"]
Nov 03 16:39:48.338833 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.338760    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-data\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.339044 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.338904    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istiod-ca-cert\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.339044 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.338951    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-workload-certs\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.339166 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.339054    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-credential-socket\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.339166 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.339121    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-envoy\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.339274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.339162    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-podinfo\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.339274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.339236    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-workload-socket\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.339274 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.339263    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-995g7\" (UniqueName: \"kubernetes.io/projected/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-kube-api-access-995g7\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.339446 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.339297    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-token\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.361784 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.361744    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:48.440182 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.440148    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-data\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.440353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.440193    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istiod-ca-cert\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.440353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.440231    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-workload-certs\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.440353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.440270    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-credential-socket\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.440353 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.440326    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-envoy\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.440590 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.440359    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-podinfo\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.440590 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.440408    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-workload-socket\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.440590 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.440432    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-995g7\" (UniqueName: \"kubernetes.io/projected/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-kube-api-access-995g7\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.440590 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.440482    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-token\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.440827 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.440604    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-data\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.440827 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.440672    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-workload-certs\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.440827 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.440743    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-credential-socket\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.440827 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.440758    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-workload-socket\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.441145 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.441123    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istiod-ca-cert\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.442845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.442820    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-envoy\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.443057 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.443041    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-podinfo\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.452786 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.452761    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-token\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.452927 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.452909    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-995g7\" (UniqueName: \"kubernetes.io/projected/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-kube-api-access-995g7\") pod \"e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") " pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.541079 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.541038    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:48.558260 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.558231    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"]
Nov 03 16:39:48.558487 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.558449    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl" podUID="8687a740-c507-4497-ae4a-60bb890a4070" containerName="maas-api" containerID="cri-o://1550a67faf45b34373874992a9bb473f2fc6d1391a0c2540b774484a06282ae4" gracePeriod=30
Nov 03 16:39:48.563930 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.563896    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:48.564040 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.564015    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:48.605836 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.605811    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"]
Nov 03 16:39:48.610562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.610536    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:48.623087 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.621838    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"]
Nov 03 16:39:48.701593 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.701555    6506 generic.go:356] "Generic (PLEG): container finished" podID="8687a740-c507-4497-ae4a-60bb890a4070" containerID="1550a67faf45b34373874992a9bb473f2fc6d1391a0c2540b774484a06282ae4" exitCode=0
Nov 03 16:39:48.701749 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.701631    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl" event={"ID":"8687a740-c507-4497-ae4a-60bb890a4070","Type":"ContainerDied","Data":"1550a67faf45b34373874992a9bb473f2fc6d1391a0c2540b774484a06282ae4"}
Nov 03 16:39:48.743451 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.743417    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f56eed-22c8-42d9-b4a6-8ad4d6e1d308-metrics-certs\") pod \"maas-api-55668dcc6b-9bt9n\" (UID: \"33f56eed-22c8-42d9-b4a6-8ad4d6e1d308\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:48.743593 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.743463    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f56eed-22c8-42d9-b4a6-8ad4d6e1d308-openshift-service-ca\") pod \"maas-api-55668dcc6b-9bt9n\" (UID: \"33f56eed-22c8-42d9-b4a6-8ad4d6e1d308\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:48.743593 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.743527    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5rfq\" (UniqueName: \"kubernetes.io/projected/33f56eed-22c8-42d9-b4a6-8ad4d6e1d308-kube-api-access-j5rfq\") pod \"maas-api-55668dcc6b-9bt9n\" (UID: \"33f56eed-22c8-42d9-b4a6-8ad4d6e1d308\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:48.743593 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.743588    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f56eed-22c8-42d9-b4a6-8ad4d6e1d308-maas-api-tls\") pod \"maas-api-55668dcc6b-9bt9n\" (UID: \"33f56eed-22c8-42d9-b4a6-8ad4d6e1d308\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:48.798020 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.797962    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:48.845256 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.845224    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f56eed-22c8-42d9-b4a6-8ad4d6e1d308-metrics-certs\") pod \"maas-api-55668dcc6b-9bt9n\" (UID: \"33f56eed-22c8-42d9-b4a6-8ad4d6e1d308\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:48.845401 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.845267    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f56eed-22c8-42d9-b4a6-8ad4d6e1d308-openshift-service-ca\") pod \"maas-api-55668dcc6b-9bt9n\" (UID: \"33f56eed-22c8-42d9-b4a6-8ad4d6e1d308\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:48.845401 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.845293    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5rfq\" (UniqueName: \"kubernetes.io/projected/33f56eed-22c8-42d9-b4a6-8ad4d6e1d308-kube-api-access-j5rfq\") pod \"maas-api-55668dcc6b-9bt9n\" (UID: \"33f56eed-22c8-42d9-b4a6-8ad4d6e1d308\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:48.845401 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.845381    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f56eed-22c8-42d9-b4a6-8ad4d6e1d308-maas-api-tls\") pod \"maas-api-55668dcc6b-9bt9n\" (UID: \"33f56eed-22c8-42d9-b4a6-8ad4d6e1d308\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:48.845904 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.845880    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f56eed-22c8-42d9-b4a6-8ad4d6e1d308-openshift-service-ca\") pod \"maas-api-55668dcc6b-9bt9n\" (UID: \"33f56eed-22c8-42d9-b4a6-8ad4d6e1d308\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:48.847748 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.847730    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f56eed-22c8-42d9-b4a6-8ad4d6e1d308-metrics-certs\") pod \"maas-api-55668dcc6b-9bt9n\" (UID: \"33f56eed-22c8-42d9-b4a6-8ad4d6e1d308\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:48.847847 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.847766    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f56eed-22c8-42d9-b4a6-8ad4d6e1d308-maas-api-tls\") pod \"maas-api-55668dcc6b-9bt9n\" (UID: \"33f56eed-22c8-42d9-b4a6-8ad4d6e1d308\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:48.857593 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.857572    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5rfq\" (UniqueName: \"kubernetes.io/projected/33f56eed-22c8-42d9-b4a6-8ad4d6e1d308-kube-api-access-j5rfq\") pod \"maas-api-55668dcc6b-9bt9n\" (UID: \"33f56eed-22c8-42d9-b4a6-8ad4d6e1d308\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:48.869308 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.869286    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"]
Nov 03 16:39:48.870819 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:39:48.870780    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe18cde_0f96_486e_b4c0_8e1d7aa15acc.slice/crio-fb77ed36fd3f2eed3537e29d7e17169865479bcea4437778bc806c67ffd91aa1 WatchSource:0}: Error finding container fb77ed36fd3f2eed3537e29d7e17169865479bcea4437778bc806c67ffd91aa1: Status 404 returned error can't find the container with id fb77ed36fd3f2eed3537e29d7e17169865479bcea4437778bc806c67ffd91aa1
Nov 03 16:39:48.872927 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.872902    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:39:48.872996 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.872956    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:39:48.872996 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.872984    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:39:48.939410 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.939382    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:48.946673 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.946641    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8687a740-c507-4497-ae4a-60bb890a4070-kube-api-access-bf9b4\" (UniqueName: \"kubernetes.io/projected/8687a740-c507-4497-ae4a-60bb890a4070-kube-api-access-bf9b4\") pod \"8687a740-c507-4497-ae4a-60bb890a4070\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") "
Nov 03 16:39:48.946865 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.946723    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8687a740-c507-4497-ae4a-60bb890a4070-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8687a740-c507-4497-ae4a-60bb890a4070-metrics-certs\") pod \"8687a740-c507-4497-ae4a-60bb890a4070\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") "
Nov 03 16:39:48.946865 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.946759    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8687a740-c507-4497-ae4a-60bb890a4070-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8687a740-c507-4497-ae4a-60bb890a4070-openshift-service-ca\") pod \"8687a740-c507-4497-ae4a-60bb890a4070\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") "
Nov 03 16:39:48.946865 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.946839    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8687a740-c507-4497-ae4a-60bb890a4070-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8687a740-c507-4497-ae4a-60bb890a4070-maas-api-tls\") pod \"8687a740-c507-4497-ae4a-60bb890a4070\" (UID: \"8687a740-c507-4497-ae4a-60bb890a4070\") "
Nov 03 16:39:48.947282 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.947210    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8687a740-c507-4497-ae4a-60bb890a4070-openshift-service-ca" pod "8687a740-c507-4497-ae4a-60bb890a4070" (UID: "8687a740-c507-4497-ae4a-60bb890a4070"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:39:48.949677 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.949656    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8687a740-c507-4497-ae4a-60bb890a4070-maas-api-tls" pod "8687a740-c507-4497-ae4a-60bb890a4070" (UID: "8687a740-c507-4497-ae4a-60bb890a4070"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:39:48.949786 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.949762    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8687a740-c507-4497-ae4a-60bb890a4070-kube-api-access-bf9b4" pod "8687a740-c507-4497-ae4a-60bb890a4070" (UID: "8687a740-c507-4497-ae4a-60bb890a4070"). InnerVolumeSpecName "kube-api-access-bf9b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:39:48.949960 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:48.949932    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8687a740-c507-4497-ae4a-60bb890a4070-metrics-certs" pod "8687a740-c507-4497-ae4a-60bb890a4070" (UID: "8687a740-c507-4497-ae4a-60bb890a4070"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:39:49.048744 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.048675    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf9b4\" (UniqueName: \"kubernetes.io/projected/8687a740-c507-4497-ae4a-60bb890a4070-kube-api-access-bf9b4\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:49.048744 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.048702    6506 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8687a740-c507-4497-ae4a-60bb890a4070-metrics-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:49.048744 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.048713    6506 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8687a740-c507-4497-ae4a-60bb890a4070-openshift-service-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:49.048744 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.048722    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8687a740-c507-4497-ae4a-60bb890a4070-maas-api-tls\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:49.223179 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.223145    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"]
Nov 03 16:39:49.709395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.709300    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt" event={"ID":"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc","Type":"ContainerStarted","Data":"17d8e0ae6f05e6bbb7da8ba425ccd620cbdfc4dbff11c940810db0b2b1983a14"}
Nov 03 16:39:49.709395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.709357    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt" event={"ID":"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc","Type":"ContainerStarted","Data":"fb77ed36fd3f2eed3537e29d7e17169865479bcea4437778bc806c67ffd91aa1"}
Nov 03 16:39:49.710930 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.710905    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"
Nov 03 16:39:49.711053 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.710931    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl" event={"ID":"8687a740-c507-4497-ae4a-60bb890a4070","Type":"ContainerDied","Data":"5bd5231ffd02664ce003af0a7f8fc8a5e7818e70ec460033557faaa685161bd4"}
Nov 03 16:39:49.711053 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.710986    6506 scope.go:122] "RemoveContainer" containerID="1550a67faf45b34373874992a9bb473f2fc6d1391a0c2540b774484a06282ae4"
Nov 03 16:39:49.716020 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.715992    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n" event={"ID":"33f56eed-22c8-42d9-b4a6-8ad4d6e1d308","Type":"ContainerStarted","Data":"aaa22c551f53e2bfa4cb179eabbbf73492e27500344dc0d4720993d4d6b62538"}
Nov 03 16:39:49.716117 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.716044    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n" event={"ID":"33f56eed-22c8-42d9-b4a6-8ad4d6e1d308","Type":"ContainerStarted","Data":"5a279089064783967a8c1658230e71c5e648f98adb74484d0a4a8c888c1d79d7"}
Nov 03 16:39:49.716199 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.716132    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:49.743448 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.743405    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt" podStartSLOduration=1.743391637 podStartE2EDuration="1.743391637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:49.742834189 +0000 UTC m=+2203.756232333" watchObservedRunningTime="2026-11-03 16:39:49.743391637 +0000 UTC m=+2203.756789783"
Nov 03 16:39:49.760674 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.760650    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"]
Nov 03 16:39:49.764306 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.764285    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-87dc4db6-n5pvl"]
Nov 03 16:39:49.783057 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:49.783019    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n" podStartSLOduration=1.382223952 podStartE2EDuration="1.783006732s" totalImagesPullingTime="400.78278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:49.782629923 +0000 UTC m=+2203.796028070" watchObservedRunningTime="2026-11-03 16:39:49.783006732 +0000 UTC m=+2203.796404877"
Nov 03 16:39:50.101888 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:50.101854    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8687a740-c507-4497-ae4a-60bb890a4070" path="/var/lib/kubelet/pods/8687a740-c507-4497-ae4a-60bb890a4070/volumes"
Nov 03 16:39:50.564441 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:50.564414    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:50.569364 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:50.569342    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:50.723018 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:50.722993    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:50.723964 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:50.723941    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:39:51.334482 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:51.334455    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:51.515462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:51.515405    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:52.517905 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.517866    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:52.627461 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.627424    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"]
Nov 03 16:39:52.628359 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.627999    6506 cpu_manager.go:414] "containerMap: removing container" podUID="8687a740-c507-4497-ae4a-60bb890a4070" containerName="maas-api"
Nov 03 16:39:52.628359 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.628025    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8687a740-c507-4497-ae4a-60bb890a4070" containerName="maas-api"
Nov 03 16:39:52.628359 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.628176    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="8687a740-c507-4497-ae4a-60bb890a4070" containerName="maas-api"
Nov 03 16:39:52.632669 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.632652    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:52.686045 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.685450    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-56d83be3-serving-cert\""
Nov 03 16:39:52.687238 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.687196    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"]
Nov 03 16:39:52.696050 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.695631    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-maas-api-tls\") pod \"maas-api-e2e-ait-56d83be3-5868748cb7-874fr\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:52.696050 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.695705    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-metrics-certs\") pod \"maas-api-e2e-ait-56d83be3-5868748cb7-874fr\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:52.696050 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.695888    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:52.696050 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.695925    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw45d\" (UniqueName: \"kubernetes.io/projected/3c10a338-da30-4408-bf71-a207b8a851b3-kube-api-access-hw45d\") pod \"maas-api-e2e-ait-56d83be3-5868748cb7-874fr\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:52.696050 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.695969    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c10a338-da30-4408-bf71-a207b8a851b3-openshift-service-ca\") pod \"maas-api-e2e-ait-56d83be3-5868748cb7-874fr\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:52.796604 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.796571    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw45d\" (UniqueName: \"kubernetes.io/projected/3c10a338-da30-4408-bf71-a207b8a851b3-kube-api-access-hw45d\") pod \"maas-api-e2e-ait-56d83be3-5868748cb7-874fr\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:52.796604 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.796612    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c10a338-da30-4408-bf71-a207b8a851b3-openshift-service-ca\") pod \"maas-api-e2e-ait-56d83be3-5868748cb7-874fr\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:52.796919 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.796775    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-maas-api-tls\") pod \"maas-api-e2e-ait-56d83be3-5868748cb7-874fr\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:52.796919 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.796884    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-metrics-certs\") pod \"maas-api-e2e-ait-56d83be3-5868748cb7-874fr\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:52.797051 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:39:52.796976    6506 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-serving-cert: secret "maas-api-e2e-ait-56d83be3-serving-cert" not found
Nov 03 16:39:52.797114 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:39:52.797053    6506 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-maas-api-tls podName:3c10a338-da30-4408-bf71-a207b8a851b3 nodeName:}" failed. No retries permitted until 2026-11-03 16:39:53.297028543 +0000 UTC m=+2207.310426683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-maas-api-tls") pod "maas-api-e2e-ait-56d83be3-5868748cb7-874fr" (UID: "3c10a338-da30-4408-bf71-a207b8a851b3") : secret "maas-api-e2e-ait-56d83be3-serving-cert" not found
Nov 03 16:39:52.797423 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.797393    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c10a338-da30-4408-bf71-a207b8a851b3-openshift-service-ca\") pod \"maas-api-e2e-ait-56d83be3-5868748cb7-874fr\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:52.799618 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.799599    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-metrics-certs\") pod \"maas-api-e2e-ait-56d83be3-5868748cb7-874fr\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:52.820987 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.820962    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw45d\" (UniqueName: \"kubernetes.io/projected/3c10a338-da30-4408-bf71-a207b8a851b3-kube-api-access-hw45d\") pod \"maas-api-e2e-ait-56d83be3-5868748cb7-874fr\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:52.854873 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:52.854844    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:53.020009 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:53.019973    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:53.302013 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:53.301978    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-maas-api-tls\") pod \"maas-api-e2e-ait-56d83be3-5868748cb7-874fr\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:53.304487 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:53.304465    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-maas-api-tls\") pod \"maas-api-e2e-ait-56d83be3-5868748cb7-874fr\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:53.548675 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:53.548645    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:53.632184 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:53.632148    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:53.829410 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:53.829387    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"]
Nov 03 16:39:53.830067 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:39:53.830040    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c10a338_da30_4408_bf71_a207b8a851b3.slice/crio-a934040b50b2a889ab679bb7a31df0a302db6784b2b438293cd1eb0b8b87d620 WatchSource:0}: Error finding container a934040b50b2a889ab679bb7a31df0a302db6784b2b438293cd1eb0b8b87d620: Status 404 returned error can't find the container with id a934040b50b2a889ab679bb7a31df0a302db6784b2b438293cd1eb0b8b87d620
Nov 03 16:39:54.626718 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:54.626686    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:54.745657 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:54.745616    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr" event={"ID":"3c10a338-da30-4408-bf71-a207b8a851b3","Type":"ContainerStarted","Data":"256f38000314093d52701a9fd36272105b1a7c2743a045539b79cf6e5d925128"}
Nov 03 16:39:54.745818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:54.745670    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr" event={"ID":"3c10a338-da30-4408-bf71-a207b8a851b3","Type":"ContainerStarted","Data":"a934040b50b2a889ab679bb7a31df0a302db6784b2b438293cd1eb0b8b87d620"}
Nov 03 16:39:54.745818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:54.745779    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:39:54.769278 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:54.769229    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr" podStartSLOduration=2.390675932 podStartE2EDuration="2.7692155s" totalImagesPullingTime="378.539568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:54.767198263 +0000 UTC m=+2208.780596410" watchObservedRunningTime="2026-11-03 16:39:54.7692155 +0000 UTC m=+2208.782613646"
Nov 03 16:39:55.497634 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:55.497594    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:55.735905 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:55.735873    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55668dcc6b-9bt9n"
Nov 03 16:39:56.224821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:56.224157    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:56.907001 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:56.906930    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:57.643506 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:57.643464    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:58.420819 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:58.413761    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:58.795891 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:39:58.795857    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:39:59.191375 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:59.191278    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:39:59.905588 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:39:59.905555    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:00.629714 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:00.629662    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:00.760608 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:00.760580    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:40:01.382040 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:01.382007    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:02.192554 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:02.192522    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:02.899256 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:02.899215    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:03.703537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:03.703503    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:04.540744 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:04.540714    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:04.808551 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:04.808517    6506 generic.go:356] "Generic (PLEG): container finished" podID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" containerID="17d8e0ae6f05e6bbb7da8ba425ccd620cbdfc4dbff11c940810db0b2b1983a14" exitCode=0
Nov 03 16:40:04.808723 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:04.808573    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt" event={"ID":"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc","Type":"ContainerDied","Data":"17d8e0ae6f05e6bbb7da8ba425ccd620cbdfc4dbff11c940810db0b2b1983a14"}
Nov 03 16:40:04.809029 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:04.809013    6506 scope.go:122] "RemoveContainer" containerID="17d8e0ae6f05e6bbb7da8ba425ccd620cbdfc4dbff11c940810db0b2b1983a14"
Nov 03 16:40:04.810367 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:04.810338    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:40:04.810440 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:04.810405    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:40:04.810440 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:04.810435    6506 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 16:40:05.331599 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:05.331559    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:05.732032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:05.731930    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:40:05.837154 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:05.837119    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt" event={"ID":"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc","Type":"ContainerStarted","Data":"43c89b8ecbd1f0c78f8e8e36634c14212ee0fd92d4497583cf730bf6d69edcd7"}
Nov 03 16:40:06.135657 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:06.135617    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:06.564577 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:06.564543    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:40:06.565865 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:06.565842    6506 patch_prober.go:29] interesting pod/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 16:40:06.565939 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:06.565890    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt" podUID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 16:40:06.838576 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:06.838477    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:07.564994 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:07.564960    6506 patch_prober.go:29] interesting pod/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 16:40:07.565189 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:07.565012    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt" podUID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 16:40:07.699442 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:07.699408    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:08.527903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:08.527867    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:08.564466 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:08.564431    6506 patch_prober.go:29] interesting pod/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 16:40:08.564628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:08.564520    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt" podUID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 16:40:09.425678 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:09.425610    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:09.565187 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:09.565146    6506 patch_prober.go:29] interesting pod/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 16:40:09.565578 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:09.565211    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt" podUID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 16:40:10.282765 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:10.282731    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:10.564870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:10.564767    6506 patch_prober.go:29] interesting pod/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 16:40:10.564870 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:10.564850    6506 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt" podUID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 16:40:10.867641 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:10.867513    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:11.054247 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:11.054213    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:11.569144 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:11.569116    6506 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:40:11.569409 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:11.569393    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:40:11.570129 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:11.570108    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:40:12.021000 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:12.020964    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:12.795118 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:12.795054    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:13.519881 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:13.519845    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:14.302265 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:14.302230    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:15.012728 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:15.012697    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:15.738067 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:15.738011    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:16.461436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:16.461394    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:17.124128 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:17.124088    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:17.496843 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:17.496693    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:17.656476 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:17.656444    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:17.694967 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:17.694928    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"]
Nov 03 16:40:17.695279 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:17.695207    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr" podUID="3c10a338-da30-4408-bf71-a207b8a851b3" containerName="maas-api" containerID="cri-o://256f38000314093d52701a9fd36272105b1a7c2743a045539b79cf6e5d925128" gracePeriod=30
Nov 03 16:40:17.925622 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:17.925589    6506 generic.go:356] "Generic (PLEG): container finished" podID="3c10a338-da30-4408-bf71-a207b8a851b3" containerID="256f38000314093d52701a9fd36272105b1a7c2743a045539b79cf6e5d925128" exitCode=0
Nov 03 16:40:17.925809 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:17.925673    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr" event={"ID":"3c10a338-da30-4408-bf71-a207b8a851b3","Type":"ContainerDied","Data":"256f38000314093d52701a9fd36272105b1a7c2743a045539b79cf6e5d925128"}
Nov 03 16:40:17.974040 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:17.974016    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:40:18.068127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.068052    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-maas-api-tls\") pod \"3c10a338-da30-4408-bf71-a207b8a851b3\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") "
Nov 03 16:40:18.068127 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.068100    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-metrics-certs\") pod \"3c10a338-da30-4408-bf71-a207b8a851b3\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") "
Nov 03 16:40:18.068311 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.068248    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c10a338-da30-4408-bf71-a207b8a851b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c10a338-da30-4408-bf71-a207b8a851b3-openshift-service-ca\") pod \"3c10a338-da30-4408-bf71-a207b8a851b3\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") "
Nov 03 16:40:18.068311 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.068283    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c10a338-da30-4408-bf71-a207b8a851b3-kube-api-access-hw45d\" (UniqueName: \"kubernetes.io/projected/3c10a338-da30-4408-bf71-a207b8a851b3-kube-api-access-hw45d\") pod \"3c10a338-da30-4408-bf71-a207b8a851b3\" (UID: \"3c10a338-da30-4408-bf71-a207b8a851b3\") "
Nov 03 16:40:18.068572 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.068545    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c10a338-da30-4408-bf71-a207b8a851b3-openshift-service-ca" pod "3c10a338-da30-4408-bf71-a207b8a851b3" (UID: "3c10a338-da30-4408-bf71-a207b8a851b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:40:18.068689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.068623    6506 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c10a338-da30-4408-bf71-a207b8a851b3-openshift-service-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:18.070300 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.070280    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-maas-api-tls" pod "3c10a338-da30-4408-bf71-a207b8a851b3" (UID: "3c10a338-da30-4408-bf71-a207b8a851b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:40:18.070462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.070426    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c10a338-da30-4408-bf71-a207b8a851b3-kube-api-access-hw45d" pod "3c10a338-da30-4408-bf71-a207b8a851b3" (UID: "3c10a338-da30-4408-bf71-a207b8a851b3"). InnerVolumeSpecName "kube-api-access-hw45d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:40:18.070556 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.070483    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-metrics-certs" pod "3c10a338-da30-4408-bf71-a207b8a851b3" (UID: "3c10a338-da30-4408-bf71-a207b8a851b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:40:18.082027 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.081998    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:18.169564 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.169519    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-maas-api-tls\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:18.169564 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.169556    6506 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c10a338-da30-4408-bf71-a207b8a851b3-metrics-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:18.169564 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.169569    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw45d\" (UniqueName: \"kubernetes.io/projected/3c10a338-da30-4408-bf71-a207b8a851b3-kube-api-access-hw45d\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:18.237447 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.237412    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:18.931878 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.931842    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr" event={"ID":"3c10a338-da30-4408-bf71-a207b8a851b3","Type":"ContainerDied","Data":"a934040b50b2a889ab679bb7a31df0a302db6784b2b438293cd1eb0b8b87d620"}
Nov 03 16:40:18.932309 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.931908    6506 scope.go:122] "RemoveContainer" containerID="256f38000314093d52701a9fd36272105b1a7c2743a045539b79cf6e5d925128"
Nov 03 16:40:18.932309 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.931853    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"
Nov 03 16:40:18.952347 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.952313    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"]
Nov 03 16:40:18.959850 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:18.959823    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56d83be3-5868748cb7-874fr"]
Nov 03 16:40:20.102405 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:20.102371    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c10a338-da30-4408-bf71-a207b8a851b3" path="/var/lib/kubelet/pods/3c10a338-da30-4408-bf71-a207b8a851b3/volumes"
Nov 03 16:40:23.495645 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:23.485874    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"]
Nov 03 16:40:23.495645 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:23.486207    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt" podUID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" containerName="istio-proxy" containerID="cri-o://43c89b8ecbd1f0c78f8e8e36634c14212ee0fd92d4497583cf730bf6d69edcd7" gracePeriod=30
Nov 03 16:40:23.595612 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:23.595570    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:24.867603 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:24.867543    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:25.340269 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:25.340238    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:25.487948 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:25.487901    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:26.572021 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:26.571988    6506 patch_prober.go:29] interesting pod/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:40:26.572396 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:26.572042    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt" podUID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:40:28.743971 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.743935    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:40:28.797839 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:40:28.797806    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:40:28.875360 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.875324    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-token\" (UniqueName: \"kubernetes.io/projected/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-token\") pod \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") "
Nov 03 16:40:28.875543 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.875379    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-kube-api-access-995g7\" (UniqueName: \"kubernetes.io/projected/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-kube-api-access-995g7\") pod \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") "
Nov 03 16:40:28.875543 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.875435    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-credential-socket\") pod \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") "
Nov 03 16:40:28.875543 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.875484    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-workload-certs\") pod \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") "
Nov 03 16:40:28.875543 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.875512    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istiod-ca-cert\") pod \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") "
Nov 03 16:40:28.875756 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.875608    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-envoy\") pod \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") "
Nov 03 16:40:28.875756 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.875679    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-workload-socket\") pod \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") "
Nov 03 16:40:28.875756 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.875739    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-podinfo\") pod \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") "
Nov 03 16:40:28.875951 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.875772    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-data\") pod \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\" (UID: \"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc\") "
Nov 03 16:40:28.875951 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.875790    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-credential-socket" pod "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" (UID: "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:40:28.875951 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.875780    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-workload-certs" pod "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" (UID: "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:40:28.876098 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.876018    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-workload-socket" pod "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" (UID: "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:40:28.876286 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.876263    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-data" pod "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" (UID: "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:40:28.876463 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.876434    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istiod-ca-cert" pod "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" (UID: "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:40:28.876608 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.876593    6506 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-data\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:28.876681 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.876612    6506 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-credential-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:28.876681 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.876644    6506 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-workload-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:28.876681 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.876660    6506 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istiod-ca-cert\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:28.876681 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.876675    6506 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-workload-socket\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:28.877857 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.877833    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-token" pod "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" (UID: "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:40:28.877951 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.877879    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-envoy" pod "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" (UID: "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:40:28.877951 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.877882    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-kube-api-access-995g7" pod "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" (UID: "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc"). InnerVolumeSpecName "kube-api-access-995g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:40:28.878027 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.877946    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-podinfo" pod "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" (UID: "ffe18cde-0f96-486e-b4c0-8e1d7aa15acc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:40:28.977687 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.977652    6506 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-envoy\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:28.977894 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.977712    6506 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-podinfo\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:28.977894 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.977725    6506 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-istio-token\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:28.977894 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.977753    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-995g7\" (UniqueName: \"kubernetes.io/projected/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc-kube-api-access-995g7\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:28.989113 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.989088    6506 generic.go:356] "Generic (PLEG): container finished" podID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" containerID="43c89b8ecbd1f0c78f8e8e36634c14212ee0fd92d4497583cf730bf6d69edcd7" exitCode=0
Nov 03 16:40:28.989244 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.989129    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt" event={"ID":"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc","Type":"ContainerDied","Data":"43c89b8ecbd1f0c78f8e8e36634c14212ee0fd92d4497583cf730bf6d69edcd7"}
Nov 03 16:40:28.989244 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.989190    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt" event={"ID":"ffe18cde-0f96-486e-b4c0-8e1d7aa15acc","Type":"ContainerDied","Data":"fb77ed36fd3f2eed3537e29d7e17169865479bcea4437778bc806c67ffd91aa1"}
Nov 03 16:40:28.989244 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.989217    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"
Nov 03 16:40:28.989368 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.989222    6506 scope.go:122] "RemoveContainer" containerID="43c89b8ecbd1f0c78f8e8e36634c14212ee0fd92d4497583cf730bf6d69edcd7"
Nov 03 16:40:28.999302 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:28.999275    6506 scope.go:122] "RemoveContainer" containerID="17d8e0ae6f05e6bbb7da8ba425ccd620cbdfc4dbff11c940810db0b2b1983a14"
Nov 03 16:40:29.008016 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:29.007997    6506 scope.go:122] "RemoveContainer" containerID="43c89b8ecbd1f0c78f8e8e36634c14212ee0fd92d4497583cf730bf6d69edcd7"
Nov 03 16:40:29.008301 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:40:29.008277    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43c89b8ecbd1f0c78f8e8e36634c14212ee0fd92d4497583cf730bf6d69edcd7\": container with ID starting with 43c89b8ecbd1f0c78f8e8e36634c14212ee0fd92d4497583cf730bf6d69edcd7 not found: ID does not exist" containerID="43c89b8ecbd1f0c78f8e8e36634c14212ee0fd92d4497583cf730bf6d69edcd7"
Nov 03 16:40:29.008414 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:29.008309    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43c89b8ecbd1f0c78f8e8e36634c14212ee0fd92d4497583cf730bf6d69edcd7"} err="failed to get container status \"43c89b8ecbd1f0c78f8e8e36634c14212ee0fd92d4497583cf730bf6d69edcd7\": rpc error: code = NotFound desc = could not find container \"43c89b8ecbd1f0c78f8e8e36634c14212ee0fd92d4497583cf730bf6d69edcd7\": container with ID starting with 43c89b8ecbd1f0c78f8e8e36634c14212ee0fd92d4497583cf730bf6d69edcd7 not found: ID does not exist"
Nov 03 16:40:29.008414 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:29.008325    6506 scope.go:122] "RemoveContainer" containerID="17d8e0ae6f05e6bbb7da8ba425ccd620cbdfc4dbff11c940810db0b2b1983a14"
Nov 03 16:40:29.008600 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:40:29.008572    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d8e0ae6f05e6bbb7da8ba425ccd620cbdfc4dbff11c940810db0b2b1983a14\": container with ID starting with 17d8e0ae6f05e6bbb7da8ba425ccd620cbdfc4dbff11c940810db0b2b1983a14 not found: ID does not exist" containerID="17d8e0ae6f05e6bbb7da8ba425ccd620cbdfc4dbff11c940810db0b2b1983a14"
Nov 03 16:40:29.008666 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:29.008603    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d8e0ae6f05e6bbb7da8ba425ccd620cbdfc4dbff11c940810db0b2b1983a14"} err="failed to get container status \"17d8e0ae6f05e6bbb7da8ba425ccd620cbdfc4dbff11c940810db0b2b1983a14\": rpc error: code = NotFound desc = could not find container \"17d8e0ae6f05e6bbb7da8ba425ccd620cbdfc4dbff11c940810db0b2b1983a14\": container with ID starting with 17d8e0ae6f05e6bbb7da8ba425ccd620cbdfc4dbff11c940810db0b2b1983a14 not found: ID does not exist"
Nov 03 16:40:29.040220 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:29.040173    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"]
Nov 03 16:40:29.040377 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:29.040231    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-56d83be3-openshift-default-d949dbcf7-x42kt"]
Nov 03 16:40:29.051341 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:29.051282    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:29.237291 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:29.237256    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:30.102550 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.102516    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" path="/var/lib/kubelet/pods/ffe18cde-0f96-486e-b4c0-8e1d7aa15acc/volumes"
Nov 03 16:40:30.325261 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.325217    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"]
Nov 03 16:40:30.325625 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.325609    6506 cpu_manager.go:414] "containerMap: removing container" podUID="3c10a338-da30-4408-bf71-a207b8a851b3" containerName="maas-api"
Nov 03 16:40:30.325625 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.325624    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c10a338-da30-4408-bf71-a207b8a851b3" containerName="maas-api"
Nov 03 16:40:30.325832 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.325639    6506 cpu_manager.go:414] "containerMap: removing container" podUID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" containerName="istio-proxy"
Nov 03 16:40:30.325832 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.325644    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" containerName="istio-proxy"
Nov 03 16:40:30.325832 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.325662    6506 cpu_manager.go:414] "containerMap: removing container" podUID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" containerName="istio-proxy"
Nov 03 16:40:30.325832 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.325667    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" containerName="istio-proxy"
Nov 03 16:40:30.325832 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.325728    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" containerName="istio-proxy"
Nov 03 16:40:30.325832 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.325737    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe18cde-0f96-486e-b4c0-8e1d7aa15acc" containerName="istio-proxy"
Nov 03 16:40:30.325832 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.325748    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c10a338-da30-4408-bf71-a207b8a851b3" containerName="maas-api"
Nov 03 16:40:30.328743 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.328725    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:30.338597 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.338580    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-05bcf4-serving-cert\""
Nov 03 16:40:30.344827 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.344788    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"]
Nov 03 16:40:30.479202 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.479102    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:30.490421 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.490395    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:30.490571 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.490451    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:30.490571 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.490472    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-metrics-certs\") pod \"maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:30.490644 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.490606    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl7rj\" (UniqueName: \"kubernetes.io/projected/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-kube-api-access-tl7rj\") pod \"maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:30.592114 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.592078    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl7rj\" (UniqueName: \"kubernetes.io/projected/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-kube-api-access-tl7rj\") pod \"maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:30.592295 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.592156    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:30.592295 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.592217    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:30.592295 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.592243    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-metrics-certs\") pod \"maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:30.593248 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.593176    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:30.595779 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.595753    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-metrics-certs\") pod \"maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:30.605543 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.605511    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl7rj\" (UniqueName: \"kubernetes.io/projected/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-kube-api-access-tl7rj\") pod \"maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:30.613969 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.613936    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:30.644407 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.644379    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:30.756845 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.756754    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:30.927560 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:30.927537    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"]
Nov 03 16:40:30.929333 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:40:30.929299    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbc1b1e_96b2_498a_9cb6_40f94003e46c.slice/crio-70ef1763ad31548c217e302335542a410c7c3e7767421a7104a45f2180841aca WatchSource:0}: Error finding container 70ef1763ad31548c217e302335542a410c7c3e7767421a7104a45f2180841aca: Status 404 returned error can't find the container with id 70ef1763ad31548c217e302335542a410c7c3e7767421a7104a45f2180841aca
Nov 03 16:40:31.002105 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:31.002074    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65" event={"ID":"7bbc1b1e-96b2-498a-9cb6-40f94003e46c","Type":"ContainerStarted","Data":"70ef1763ad31548c217e302335542a410c7c3e7767421a7104a45f2180841aca"}
Nov 03 16:40:32.009242 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:32.009211    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65" event={"ID":"7bbc1b1e-96b2-498a-9cb6-40f94003e46c","Type":"ContainerStarted","Data":"93c59e4b34eaaa2ad7ce8e6e15d03010e942690e005dc1b1071636730c9e99cc"}
Nov 03 16:40:32.009663 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:32.009342    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:32.016023 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:32.015999    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:32.051512 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:32.051462    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65" podStartSLOduration=1.553699798 podStartE2EDuration="2.051447292s" totalImagesPullingTime="497.747494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:32.04908443 +0000 UTC m=+2246.062482577" watchObservedRunningTime="2026-11-03 16:40:32.051447292 +0000 UTC m=+2246.064845439"
Nov 03 16:40:32.804410 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:32.804377    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:33.652438 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:33.652402    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:34.537329 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:34.537296    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:35.378511 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:35.378407    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:36.171854 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:36.171814    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:36.970511 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:36.970476    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:37.743462 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:37.743428    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:38.020269 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:38.020245    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:40:38.695548 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:38.695515    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:39.509068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:39.509032    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:40.206139 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:40.206106    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:40.926510 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:40.926445    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:41.664907 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:41.664877    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:42.527267 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:42.527221    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:43.499743 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:43.499706    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:44.037264 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:44.037227    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:44.275338 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:44.275308    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:45.130269 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:45.130234    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:46.048379 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:46.048299    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:46.829352 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:46.829320    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:47.643908 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:47.643869    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:48.426435 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:48.426398    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:49.207164 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:49.207131    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:49.973808 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:49.973761    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:50.691898 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:50.691864    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:51.396804 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:51.396749    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:52.102488 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:52.102460    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:52.823854 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:52.823817    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:53.616049 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:53.616015    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:54.297889 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:54.297774    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:55.007032 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:55.007000    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:55.702763 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:55.702688    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:56.421500 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:56.421465    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:57.115502 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:57.115471    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:57.805134 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:57.805086    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:58.519398 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:58.519357    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:40:58.795932 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:40:58.795857    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:40:59.257624 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:40:59.257530    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:00.220691 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:00.220654    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:00.935039 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:00.935003    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:01.095095 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:01.095060    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:01.873439 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:01.873397    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:02.508419 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:02.508387    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:03.261676 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:03.261641    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:03.903936 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:03.903900    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:04.602675 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:04.602645    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:05.330992 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:05.330947    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:06.023306 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:06.023274    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:06.802421 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:06.802386    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:07.166459 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:07.166379    6506 scope.go:122] "RemoveContainer" containerID="9f92ca61469f076d2fa4e361c0794a34fb2ee2ed21d1e9783a3116dc05be0a30"
Nov 03 16:41:07.587534 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:07.587493    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:08.316356 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:08.316323    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:09.011527 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:09.011487    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:09.714562 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:09.714533    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:10.418393 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:10.418362    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:10.919395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:10.919354    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:11.105282 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:11.105239    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:11.987124 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:11.987081    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:12.693431 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:12.693387    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:13.404350 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:13.404310    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:14.123486 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:14.123451    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:14.904542 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:14.904510    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:15.763068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:15.763034    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:16.425873 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:16.425828    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:17.115846 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:17.115813    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:17.823485 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:17.823452    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:18.512150 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:18.512106    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:19.218537 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:19.218500    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:19.912304 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:19.912269    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:20.601074 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:20.601034    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:21.299951 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:21.299913    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:22.085750 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:22.085714    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:22.714088 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:22.714052    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:23.414364 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:23.414320    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:24.172216 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:24.172177    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:24.909217 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:24.909163    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:25.609938 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:25.609902    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:26.306019 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:26.305978    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:27.024500 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:27.024465    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"]
Nov 03 16:41:27.026929 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:27.026906    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-shflt"]
Nov 03 16:41:27.100913 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:27.100878    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:27.262986 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:27.262954    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:28.103386 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:28.103354    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea26cba-bb06-47e0-8247-30e61f2a5a36" path="/var/lib/kubelet/pods/7ea26cba-bb06-47e0-8247-30e61f2a5a36/volumes"
Nov 03 16:41:28.106206 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:28.106163    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:28.799642 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:41:28.799602    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:41:28.803331 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:28.803304    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:29.502457 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:29.502416    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:30.218941 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:30.218903    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:30.919753 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:30.919720    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:31.630996 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:31.630960    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:32.312447 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:32.312403    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:33.045821 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:33.045768    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:33.738846 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:33.738810    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:34.427153 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:34.427121    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:35.117113 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:35.117078    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:35.831891 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:35.831852    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:36.515771 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:36.515734    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:37.216743 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:37.216709    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:37.906950 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:37.906912    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:38.625717 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:38.625680    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:39.327631 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:39.327594    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:40.119223 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:40.119188    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:40.834949 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:40.834912    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:41.515000 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:41.514963    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:42.211258 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:42.211229    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:43.197093 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:43.197051    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:43.983514 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:43.983468    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:44.700496 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:44.700459    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:45.407245 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:45.407203    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:46.025882 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:46.025840    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"]
Nov 03 16:41:46.028176 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:46.028154    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wnhbh"]
Nov 03 16:41:46.104092 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:46.104060    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b" path="/var/lib/kubelet/pods/f1ee3fd3-6307-4b57-92aa-9ac25f13ee7b/volumes"
Nov 03 16:41:46.109393 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:46.109350    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:46.817140 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:46.817107    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:47.619366 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:47.619332    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:48.416503 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:48.416468    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:49.235930 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:49.235895    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:49.486059 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:49.485976    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:50.002887 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:50.002854    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:50.819350 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:50.819313    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:51.566500 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:51.566472    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:52.315696 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:52.315653    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:53.101395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:53.101333    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:53.812631 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:53.812595    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:54.599408 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:54.599377    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:55.295549 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:55.295518    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:55.999183 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:55.999151    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:56.699603 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:56.699565    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:57.398609 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:57.398577    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:58.111481 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:58.111419    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:58.795934 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:41:58.795900    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:41:58.938665 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:58.938631    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:41:59.791459 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:41:59.791428    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:00.499981 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:00.499946    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:01.342392 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:01.342347    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:02.067027 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:02.066996    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:02.812745 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:02.812710    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:03.605922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:03.605886    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:04.523760 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:04.523724    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:05.316689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:05.316658    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:06.122661 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:06.122624    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:06.917827 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:06.917772    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:07.742135 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:07.742102    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:08.503947 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:08.503891    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:09.200079 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:09.200039    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:09.919991 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:09.919955    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:10.611523 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:10.611485    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:10.975959 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:10.975876    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:11.332347 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:11.332307    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:12.008954 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:12.008923    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:12.824068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:12.824031    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:13.525955 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:13.525919    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:14.364257 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:14.364226    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:14.583704 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:14.583677    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:14.779715 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:14.779676    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:20.094157 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.094078    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"]
Nov 03 16:42:20.094619 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.094318    6506 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65" podUID="7bbc1b1e-96b2-498a-9cb6-40f94003e46c" containerName="maas-api" containerID="cri-o://93c59e4b34eaaa2ad7ce8e6e15d03010e942690e005dc1b1071636730c9e99cc" gracePeriod=30
Nov 03 16:42:20.205100 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.205067    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:20.349618 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.349554    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:42:20.444868 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.444833    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:20.470954 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.470925    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-kube-api-access-tl7rj\" (UniqueName: \"kubernetes.io/projected/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-kube-api-access-tl7rj\") pod \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") "
Nov 03 16:42:20.471098 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.471039    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-openshift-service-ca\") pod \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") "
Nov 03 16:42:20.471163 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.471094    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-maas-api-tls\") pod \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") "
Nov 03 16:42:20.471223 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.471162    6506 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-metrics-certs\") pod \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\" (UID: \"7bbc1b1e-96b2-498a-9cb6-40f94003e46c\") "
Nov 03 16:42:20.471450 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.471416    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-openshift-service-ca" pod "7bbc1b1e-96b2-498a-9cb6-40f94003e46c" (UID: "7bbc1b1e-96b2-498a-9cb6-40f94003e46c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:42:20.473303 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.473277    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-maas-api-tls" pod "7bbc1b1e-96b2-498a-9cb6-40f94003e46c" (UID: "7bbc1b1e-96b2-498a-9cb6-40f94003e46c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:42:20.473393 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.473333    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-kube-api-access-tl7rj" pod "7bbc1b1e-96b2-498a-9cb6-40f94003e46c" (UID: "7bbc1b1e-96b2-498a-9cb6-40f94003e46c"). InnerVolumeSpecName "kube-api-access-tl7rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:42:20.473393 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.473331    6506 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-metrics-certs" pod "7bbc1b1e-96b2-498a-9cb6-40f94003e46c" (UID: "7bbc1b1e-96b2-498a-9cb6-40f94003e46c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:42:20.572867 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.572830    6506 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-maas-api-tls\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:42:20.572867 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.572860    6506 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-metrics-certs\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:42:20.572867 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.572871    6506 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl7rj\" (UniqueName: \"kubernetes.io/projected/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-kube-api-access-tl7rj\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:42:20.573103 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.572880    6506 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbc1b1e-96b2-498a-9cb6-40f94003e46c-openshift-service-ca\") on node \"ip-10-0-137-159.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:42:20.601072 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.601007    6506 generic.go:356] "Generic (PLEG): container finished" podID="7bbc1b1e-96b2-498a-9cb6-40f94003e46c" containerID="93c59e4b34eaaa2ad7ce8e6e15d03010e942690e005dc1b1071636730c9e99cc" exitCode=0
Nov 03 16:42:20.601072 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.601055    6506 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"
Nov 03 16:42:20.601228 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.601095    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65" event={"ID":"7bbc1b1e-96b2-498a-9cb6-40f94003e46c","Type":"ContainerDied","Data":"93c59e4b34eaaa2ad7ce8e6e15d03010e942690e005dc1b1071636730c9e99cc"}
Nov 03 16:42:20.601228 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.601153    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65" event={"ID":"7bbc1b1e-96b2-498a-9cb6-40f94003e46c","Type":"ContainerDied","Data":"70ef1763ad31548c217e302335542a410c7c3e7767421a7104a45f2180841aca"}
Nov 03 16:42:20.601228 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.601190    6506 scope.go:122] "RemoveContainer" containerID="93c59e4b34eaaa2ad7ce8e6e15d03010e942690e005dc1b1071636730c9e99cc"
Nov 03 16:42:20.611397 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.611380    6506 scope.go:122] "RemoveContainer" containerID="93c59e4b34eaaa2ad7ce8e6e15d03010e942690e005dc1b1071636730c9e99cc"
Nov 03 16:42:20.611638 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:42:20.611619    6506 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c59e4b34eaaa2ad7ce8e6e15d03010e942690e005dc1b1071636730c9e99cc\": container with ID starting with 93c59e4b34eaaa2ad7ce8e6e15d03010e942690e005dc1b1071636730c9e99cc not found: ID does not exist" containerID="93c59e4b34eaaa2ad7ce8e6e15d03010e942690e005dc1b1071636730c9e99cc"
Nov 03 16:42:20.611698 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.611649    6506 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c59e4b34eaaa2ad7ce8e6e15d03010e942690e005dc1b1071636730c9e99cc"} err="failed to get container status \"93c59e4b34eaaa2ad7ce8e6e15d03010e942690e005dc1b1071636730c9e99cc\": rpc error: code = NotFound desc = could not find container \"93c59e4b34eaaa2ad7ce8e6e15d03010e942690e005dc1b1071636730c9e99cc\": container with ID starting with 93c59e4b34eaaa2ad7ce8e6e15d03010e942690e005dc1b1071636730c9e99cc not found: ID does not exist"
Nov 03 16:42:20.645368 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.645345    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"]
Nov 03 16:42:20.650603 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.650582    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-05bcf4-7cb58495fd-5tb65"]
Nov 03 16:42:20.772594 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:20.772565    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:21.034382 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:21.034350    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:22.102030 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:22.102000    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bbc1b1e-96b2-498a-9cb6-40f94003e46c" path="/var/lib/kubelet/pods/7bbc1b1e-96b2-498a-9cb6-40f94003e46c/volumes"
Nov 03 16:42:27.499467 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:27.499425    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:27.663259 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:27.663222    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:27.878681 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:27.878642    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:28.795456 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:42:28.795409    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:42:32.816924 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:32.816885    6506 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-pq785 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:42:32.817583 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:32.817548    6506 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pq785" podUID="9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:42:37.701943 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:37.701904    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:44.144747 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:44.144712    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:42:48.032130 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:48.032081    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"]
Nov 03 16:42:48.034773 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:48.034748    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a2ee2c2-6l7hv"]
Nov 03 16:42:48.103124 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:42:48.103094    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d2b6e0-f1fd-463e-bba7-4c05007df112" path="/var/lib/kubelet/pods/66d2b6e0-f1fd-463e-bba7-4c05007df112/volumes"
Nov 03 16:42:58.795690 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:42:58.795649    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:43:07.206946 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:07.206909    6506 scope.go:122] "RemoveContainer" containerID="9db2cf5877a0302f7247fb3110bf5a9cd861b5742dd425f13a446b8f72969cc8"
Nov 03 16:43:07.218903 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:07.218873    6506 scope.go:122] "RemoveContainer" containerID="3fcfe1d6270c8d31986ab7f91a6bd567753c6a45b02933862ab3152b95e913ef"
Nov 03 16:43:11.040774 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:11.040738    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:43:12.529859 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:12.529828    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:43:17.890936 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:17.890890    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59fc67b59c-ngbw5_41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e/authorino/0.log"
Nov 03 16:43:28.796530 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:43:28.796482    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:43:31.027846 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:31.027812    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-55668dcc6b-9bt9n_33f56eed-22c8-42d9-b4a6-8ad4d6e1d308/maas-api/0.log"
Nov 03 16:43:31.632703 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:31.632677    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-x2twg_f01ce01d-b2f9-45d0-b8aa-a3777b34aa65/revoke-keys/0.log"
Nov 03 16:43:31.808449 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:31.808419    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-62b2q_f0e04549-54bd-4add-bca3-5b849bfc1669/revoke-keys/0.log"
Nov 03 16:43:32.161428 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:32.161398    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cb773898-2zdw7_0f3c3d72-f12c-4d43-80ca-8170dc053d1b/revoke-keys/0.log"
Nov 03 16:43:34.024718 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:34.024666    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59fc67b59c-ngbw5_41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e/authorino/0.log"
Nov 03 16:43:35.316193 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:35.316161    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:43:35.481690 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:35.481658    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:43:36.411439 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:36.411407    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864dbfdb86-4kdzq_cbeb1f8a-f73e-4b82-bdc5-048c3e57453e/kube-auth-proxy/0.log"
Nov 03 16:43:37.155165 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:37.155136    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-9z9vz_2cf76626-1d92-4255-8343-5ae7f38a01ab/istio-proxy/1.log"
Nov 03 16:43:37.529818 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:37.529775    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-lmt2w_f9b93c18-e065-4198-897d-2d7dcdcc8d07/istio-proxy/0.log"
Nov 03 16:43:37.723992 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:37.723954    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-pq785_9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f/istio-proxy/0.log"
Nov 03 16:43:38.090684 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:38.090652    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-z5jhm_da47afac-7b95-44ce-867c-1610ca16c13a/istio-proxy/1.log"
Nov 03 16:43:39.825034 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:39.825001    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-9z9vz_2cf76626-1d92-4255-8343-5ae7f38a01ab/istio-proxy/0.log"
Nov 03 16:43:40.391002 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:40.390960    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-z5jhm_da47afac-7b95-44ce-867c-1610ca16c13a/istio-proxy/0.log"
Nov 03 16:43:41.294042 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:41.294017    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-lqsws_b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45/main/0.log"
Nov 03 16:43:41.473781 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:41.473746    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-9cw7q_993750b4-6f00-4cb4-b14e-d1ef7c7249df/main/0.log"
Nov 03 16:43:41.838573 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:41.838548    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr_1bc91de6-75b4-489b-9f16-c7175b27e41f/main/0.log"
Nov 03 16:43:42.386972 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:42.386931    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2_28df0487-be29-4efe-99d6-94b6166fa1cc/main/0.log"
Nov 03 16:43:43.266426 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:43.266394    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59fc67b59c-ngbw5_41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e/authorino/0.log"
Nov 03 16:43:54.576091 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:54.575997    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-55668dcc6b-9bt9n_33f56eed-22c8-42d9-b4a6-8ad4d6e1d308/maas-api/0.log"
Nov 03 16:43:55.024589 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:55.024554    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"]
Nov 03 16:43:55.026922 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:55.026900    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-62b2q"]
Nov 03 16:43:55.148938 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:55.148909    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-x2twg_f01ce01d-b2f9-45d0-b8aa-a3777b34aa65/revoke-keys/0.log"
Nov 03 16:43:55.695042 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:55.695010    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cb773898-2zdw7_0f3c3d72-f12c-4d43-80ca-8170dc053d1b/revoke-keys/0.log"
Nov 03 16:43:56.102474 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:56.102442    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e04549-54bd-4add-bca3-5b849bfc1669" path="/var/lib/kubelet/pods/f0e04549-54bd-4add-bca3-5b849bfc1669/volumes"
Nov 03 16:43:57.201194 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:57.201159    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59fc67b59c-ngbw5_41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e/authorino/0.log"
Nov 03 16:43:58.796352 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:43:58.796317    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:43:59.429139 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:43:59.429110    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864dbfdb86-4kdzq_cbeb1f8a-f73e-4b82-bdc5-048c3e57453e/kube-auth-proxy/0.log"
Nov 03 16:44:00.175571 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:00.175539    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-9z9vz_2cf76626-1d92-4255-8343-5ae7f38a01ab/istio-proxy/1.log"
Nov 03 16:44:00.543876 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:00.543848    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-lmt2w_f9b93c18-e065-4198-897d-2d7dcdcc8d07/istio-proxy/0.log"
Nov 03 16:44:00.722397 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:00.722367    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-pq785_9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f/istio-proxy/0.log"
Nov 03 16:44:01.111624 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:01.111591    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-z5jhm_da47afac-7b95-44ce-867c-1610ca16c13a/istio-proxy/1.log"
Nov 03 16:44:02.899612 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:02.899582    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-lqsws_b81f4c2b-2a82-4d5d-b12e-58ada0bfdb45/main/0.log"
Nov 03 16:44:03.073899 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:03.073860    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-9cw7q_993750b4-6f00-4cb4-b14e-d1ef7c7249df/main/0.log"
Nov 03 16:44:03.452694 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:03.452665    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vgtvr_1bc91de6-75b4-489b-9f16-c7175b27e41f/main/0.log"
Nov 03 16:44:04.000278 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:04.000234    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-th6q2_28df0487-be29-4efe-99d6-94b6166fa1cc/main/0.log"
Nov 03 16:44:07.249761 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:07.249730    6506 scope.go:122] "RemoveContainer" containerID="d7a2d72f8b34e80e33f00faebf90997a36ac2969a91bc9767f42f0cc86ed4afc"
Nov 03 16:44:07.258779 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:07.258756    6506 scope.go:122] "RemoveContainer" containerID="852818078bf46ab1c309f4ac2c1940b3592976ce7667c3f4bc914fbd1445f97a"
Nov 03 16:44:11.114937 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:11.114896    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:44:11.287098 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:11.287069    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:44:12.024624 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:12.024581    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"]
Nov 03 16:44:12.027640 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:12.027615    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x2twg"]
Nov 03 16:44:12.102332 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:12.102302    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01ce01d-b2f9-45d0-b8aa-a3777b34aa65" path="/var/lib/kubelet/pods/f01ce01d-b2f9-45d0-b8aa-a3777b34aa65/volumes"
Nov 03 16:44:28.795264 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:44:28.795227    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:44:32.025409 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:32.025374    6506 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"]
Nov 03 16:44:32.027905 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:32.027878    6506 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb773898-2zdw7"]
Nov 03 16:44:32.102756 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:44:32.102714    6506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3c3d72-f12c-4d43-80ca-8170dc053d1b" path="/var/lib/kubelet/pods/0f3c3d72-f12c-4d43-80ca-8170dc053d1b/volumes"
Nov 03 16:44:58.795780 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:44:58.795741    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:45:07.283306 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:07.283273    6506 scope.go:122] "RemoveContainer" containerID="56a0a2d4ffdb468f4f5660f60f2b8b0c192cdec1cc572ff84e1c46b8637f6aac"
Nov 03 16:45:07.292323 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:07.292302    6506 scope.go:122] "RemoveContainer" containerID="5ed8e2a7eff8f6e39000864dd899a54b901f3c6dfe207e82818c5e4555f7b357"
Nov 03 16:45:11.237493 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:11.237451    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:45:11.422073 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:11.422042    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:45:28.795171 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:45:28.795133    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:45:30.183309 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.183274    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v2xlg/must-gather-wph5b"]
Nov 03 16:45:30.183811 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.183692    6506 cpu_manager.go:414] "containerMap: removing container" podUID="7bbc1b1e-96b2-498a-9cb6-40f94003e46c" containerName="maas-api"
Nov 03 16:45:30.183811 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.183703    6506 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bbc1b1e-96b2-498a-9cb6-40f94003e46c" containerName="maas-api"
Nov 03 16:45:30.183811 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.183768    6506 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bbc1b1e-96b2-498a-9cb6-40f94003e46c" containerName="maas-api"
Nov 03 16:45:30.188442 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.188420    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2xlg/must-gather-wph5b"
Nov 03 16:45:30.196393 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.196338    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v2xlg\"/\"kube-root-ca.crt\""
Nov 03 16:45:30.197228 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.197205    6506 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v2xlg\"/\"openshift-service-ca.crt\""
Nov 03 16:45:30.199202 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.199183    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v2xlg/must-gather-wph5b"]
Nov 03 16:45:30.209627 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.209603    6506 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v2xlg\"/\"default-dockercfg-qmzbm\""
Nov 03 16:45:30.266940 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.266910    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:45:30.309575 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.309546    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8fl\" (UniqueName: \"kubernetes.io/projected/c888394f-9356-4821-83bc-49387d995d53-kube-api-access-2c8fl\") pod \"must-gather-wph5b\" (UID: \"c888394f-9356-4821-83bc-49387d995d53\") " pod="openshift-must-gather-v2xlg/must-gather-wph5b"
Nov 03 16:45:30.309689 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.309583    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c888394f-9356-4821-83bc-49387d995d53-must-gather-output\") pod \"must-gather-wph5b\" (UID: \"c888394f-9356-4821-83bc-49387d995d53\") " pod="openshift-must-gather-v2xlg/must-gather-wph5b"
Nov 03 16:45:30.410285 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.410250    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8fl\" (UniqueName: \"kubernetes.io/projected/c888394f-9356-4821-83bc-49387d995d53-kube-api-access-2c8fl\") pod \"must-gather-wph5b\" (UID: \"c888394f-9356-4821-83bc-49387d995d53\") " pod="openshift-must-gather-v2xlg/must-gather-wph5b"
Nov 03 16:45:30.410285 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.410289    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c888394f-9356-4821-83bc-49387d995d53-must-gather-output\") pod \"must-gather-wph5b\" (UID: \"c888394f-9356-4821-83bc-49387d995d53\") " pod="openshift-must-gather-v2xlg/must-gather-wph5b"
Nov 03 16:45:30.410664 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.410645    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c888394f-9356-4821-83bc-49387d995d53-must-gather-output\") pod \"must-gather-wph5b\" (UID: \"c888394f-9356-4821-83bc-49387d995d53\") " pod="openshift-must-gather-v2xlg/must-gather-wph5b"
Nov 03 16:45:30.440434 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.440367    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8fl\" (UniqueName: \"kubernetes.io/projected/c888394f-9356-4821-83bc-49387d995d53-kube-api-access-2c8fl\") pod \"must-gather-wph5b\" (UID: \"c888394f-9356-4821-83bc-49387d995d53\") " pod="openshift-must-gather-v2xlg/must-gather-wph5b"
Nov 03 16:45:30.504420 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.504392    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2xlg/must-gather-wph5b"
Nov 03 16:45:30.513395 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.513368    6506 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log" limit=1024
Nov 03 16:45:30.814861 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.814833    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v2xlg/must-gather-wph5b"]
Nov 03 16:45:30.818174 ip-10-0-137-159 kubenswrapper[6506]: W1103 16:45:30.818144    6506 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc888394f_9356_4821_83bc_49387d995d53.slice/crio-d0450402538b4bff1cfd30a9cce4b0cb2770bdb472a98094ea80d1ee18c30d69 WatchSource:0}: Error finding container d0450402538b4bff1cfd30a9cce4b0cb2770bdb472a98094ea80d1ee18c30d69: Status 404 returned error can't find the container with id d0450402538b4bff1cfd30a9cce4b0cb2770bdb472a98094ea80d1ee18c30d69
Nov 03 16:45:30.819936 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:30.819916    6506 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:45:31.773965 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:31.773899    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2xlg/must-gather-wph5b" event={"ID":"c888394f-9356-4821-83bc-49387d995d53","Type":"ContainerStarted","Data":"f8f7f555710232213a6a9fee7ba5f788cad43b27715a9e2d36e8ec190315d5e0"}
Nov 03 16:45:31.773965 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:31.773947    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2xlg/must-gather-wph5b" event={"ID":"c888394f-9356-4821-83bc-49387d995d53","Type":"ContainerStarted","Data":"d0450402538b4bff1cfd30a9cce4b0cb2770bdb472a98094ea80d1ee18c30d69"}
Nov 03 16:45:32.781102 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:32.781057    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2xlg/must-gather-wph5b" event={"ID":"c888394f-9356-4821-83bc-49387d995d53","Type":"ContainerStarted","Data":"bf679242859fb0f81fdeaf2a4da960b6d94fe0d9a9c0847c58e3782778299ead"}
Nov 03 16:45:33.571827 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:33.571773    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qmd98_7ea5065d-e1c6-4152-842c-833cd8155772/global-pull-secret-syncer/0.log"
Nov 03 16:45:33.881935 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:33.881841    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hdgx7_13d056b1-85f8-48eb-b9bf-aeb4b98a8170/konnectivity-agent/0.log"
Nov 03 16:45:34.056780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:34.055828    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-159.us-east-2.compute.internal_f898674fe0c94d2ff8a087a4cd76c349/haproxy/0.log"
Nov 03 16:45:41.221822 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:41.218776    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59fc67b59c-ngbw5_41bdf1a9-3a7b-47fd-b2a6-884348cf7f7e/authorino/0.log"
Nov 03 16:45:43.097628 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.097593    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-w7fdq_9c83af76-9520-4c3d-8aec-3590c97abaf7/kube-state-metrics/0.log"
Nov 03 16:45:43.130397 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.130363    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-w7fdq_9c83af76-9520-4c3d-8aec-3590c97abaf7/kube-rbac-proxy-main/0.log"
Nov 03 16:45:43.159366 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.159335    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-w7fdq_9c83af76-9520-4c3d-8aec-3590c97abaf7/kube-rbac-proxy-self/0.log"
Nov 03 16:45:43.195901 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.195872    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-56b6dcb5b8-8jqch_1e20bd59-e885-4662-bb6b-7cd403b9d752/metrics-server/0.log"
Nov 03 16:45:43.277747 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.277710    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvq8q_9f0811b4-9b91-454b-b715-6aaf519b5e92/node-exporter/0.log"
Nov 03 16:45:43.312975 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.312949    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvq8q_9f0811b4-9b91-454b-b715-6aaf519b5e92/kube-rbac-proxy/0.log"
Nov 03 16:45:43.351201 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.351134    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvq8q_9f0811b4-9b91-454b-b715-6aaf519b5e92/init-textfile/0.log"
Nov 03 16:45:43.670299 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.670222    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc/prometheus/0.log"
Nov 03 16:45:43.675205 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.675154    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-v2xlg/must-gather-wph5b" podStartSLOduration=12.838851623 podStartE2EDuration="13.675118968s" totalImagesPullingTime="836.267345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:32.803953619 +0000 UTC m=+2546.817351767" watchObservedRunningTime="2026-11-03 16:45:43.675118968 +0000 UTC m=+2557.688517156"
Nov 03 16:45:43.676026 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.676006    6506 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"]
Nov 03 16:45:43.682339 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.682316    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:43.694868 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.694788    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"]
Nov 03 16:45:43.721001 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.720956    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc/config-reloader/0.log"
Nov 03 16:45:43.748529 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.748501    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc/thanos-sidecar/0.log"
Nov 03 16:45:43.779413 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.779383    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc/kube-rbac-proxy-web/0.log"
Nov 03 16:45:43.793479 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.793446    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ddec80a2-4d74-4065-a936-beb6f3abfa10-proc\") pod \"perf-node-gather-daemonset-tq6lp\" (UID: \"ddec80a2-4d74-4065-a936-beb6f3abfa10\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:43.793672 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.793499    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddec80a2-4d74-4065-a936-beb6f3abfa10-sys\") pod \"perf-node-gather-daemonset-tq6lp\" (UID: \"ddec80a2-4d74-4065-a936-beb6f3abfa10\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:43.793672 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.793528    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjvqt\" (UniqueName: \"kubernetes.io/projected/ddec80a2-4d74-4065-a936-beb6f3abfa10-kube-api-access-cjvqt\") pod \"perf-node-gather-daemonset-tq6lp\" (UID: \"ddec80a2-4d74-4065-a936-beb6f3abfa10\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:43.793809 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.793739    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ddec80a2-4d74-4065-a936-beb6f3abfa10-lib-modules\") pod \"perf-node-gather-daemonset-tq6lp\" (UID: \"ddec80a2-4d74-4065-a936-beb6f3abfa10\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:43.793875 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.793789    6506 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ddec80a2-4d74-4065-a936-beb6f3abfa10-podres\") pod \"perf-node-gather-daemonset-tq6lp\" (UID: \"ddec80a2-4d74-4065-a936-beb6f3abfa10\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:43.826723 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.826698    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc/kube-rbac-proxy/0.log"
Nov 03 16:45:43.862299 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.862269    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc/kube-rbac-proxy-thanos/0.log"
Nov 03 16:45:43.891654 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.891600    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_838d9c9d-ad7c-45f1-a71e-dbf1b9f2e1cc/init-config-reloader/0.log"
Nov 03 16:45:43.895096 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.895064    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ddec80a2-4d74-4065-a936-beb6f3abfa10-lib-modules\") pod \"perf-node-gather-daemonset-tq6lp\" (UID: \"ddec80a2-4d74-4065-a936-beb6f3abfa10\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:43.895231 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.895115    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ddec80a2-4d74-4065-a936-beb6f3abfa10-podres\") pod \"perf-node-gather-daemonset-tq6lp\" (UID: \"ddec80a2-4d74-4065-a936-beb6f3abfa10\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:43.895231 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.895164    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ddec80a2-4d74-4065-a936-beb6f3abfa10-proc\") pod \"perf-node-gather-daemonset-tq6lp\" (UID: \"ddec80a2-4d74-4065-a936-beb6f3abfa10\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:43.895231 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.895205    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ddec80a2-4d74-4065-a936-beb6f3abfa10-lib-modules\") pod \"perf-node-gather-daemonset-tq6lp\" (UID: \"ddec80a2-4d74-4065-a936-beb6f3abfa10\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:43.895231 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.895213    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddec80a2-4d74-4065-a936-beb6f3abfa10-sys\") pod \"perf-node-gather-daemonset-tq6lp\" (UID: \"ddec80a2-4d74-4065-a936-beb6f3abfa10\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:43.895451 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.895249    6506 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjvqt\" (UniqueName: \"kubernetes.io/projected/ddec80a2-4d74-4065-a936-beb6f3abfa10-kube-api-access-cjvqt\") pod \"perf-node-gather-daemonset-tq6lp\" (UID: \"ddec80a2-4d74-4065-a936-beb6f3abfa10\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:43.895451 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.895276    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ddec80a2-4d74-4065-a936-beb6f3abfa10-proc\") pod \"perf-node-gather-daemonset-tq6lp\" (UID: \"ddec80a2-4d74-4065-a936-beb6f3abfa10\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:43.895451 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.895305    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ddec80a2-4d74-4065-a936-beb6f3abfa10-podres\") pod \"perf-node-gather-daemonset-tq6lp\" (UID: \"ddec80a2-4d74-4065-a936-beb6f3abfa10\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:43.895451 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.895400    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddec80a2-4d74-4065-a936-beb6f3abfa10-sys\") pod \"perf-node-gather-daemonset-tq6lp\" (UID: \"ddec80a2-4d74-4065-a936-beb6f3abfa10\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:43.908152 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:43.908128    6506 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjvqt\" (UniqueName: \"kubernetes.io/projected/ddec80a2-4d74-4065-a936-beb6f3abfa10-kube-api-access-cjvqt\") pod \"perf-node-gather-daemonset-tq6lp\" (UID: \"ddec80a2-4d74-4065-a936-beb6f3abfa10\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:44.001839 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:44.001724    6506 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:44.142424 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:44.142353    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7747889c6f-6psbm_cc926104-1c6c-46f3-9d8a-7f1d8394d713/thanos-query/0.log"
Nov 03 16:45:44.163995 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:44.163967    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7747889c6f-6psbm_cc926104-1c6c-46f3-9d8a-7f1d8394d713/kube-rbac-proxy-web/0.log"
Nov 03 16:45:44.193542 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:44.193501    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7747889c6f-6psbm_cc926104-1c6c-46f3-9d8a-7f1d8394d713/kube-rbac-proxy/0.log"
Nov 03 16:45:44.224732 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:44.224690    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7747889c6f-6psbm_cc926104-1c6c-46f3-9d8a-7f1d8394d713/prom-label-proxy/0.log"
Nov 03 16:45:44.259436 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:44.259348    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7747889c6f-6psbm_cc926104-1c6c-46f3-9d8a-7f1d8394d713/kube-rbac-proxy-rules/0.log"
Nov 03 16:45:44.290885 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:44.290847    6506 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"]
Nov 03 16:45:44.293022 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:44.292996    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7747889c6f-6psbm_cc926104-1c6c-46f3-9d8a-7f1d8394d713/kube-rbac-proxy-metrics/0.log"
Nov 03 16:45:44.929854 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:44.929789    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp" event={"ID":"ddec80a2-4d74-4065-a936-beb6f3abfa10","Type":"ContainerStarted","Data":"d5c1c9cbdefdeea8cca85c4ed6c2ca830a210d7c78e9ebb44a98abac44feed98"}
Nov 03 16:45:44.930068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:44.929985    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:44.930068 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:44.930036    6506 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp" event={"ID":"ddec80a2-4d74-4065-a936-beb6f3abfa10","Type":"ContainerStarted","Data":"ba14fcbd63696492073b900ee8c569dd46bea3df60a277747680af76aa24205b"}
Nov 03 16:45:44.952213 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:44.952144    6506 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp" podStartSLOduration=1.952124921 podStartE2EDuration="1.952124921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:44.951140523 +0000 UTC m=+2558.964538670" watchObservedRunningTime="2026-11-03 16:45:44.952124921 +0000 UTC m=+2558.965523069"
Nov 03 16:45:48.586620 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:48.586581    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kgdhx_5806796f-e98d-4eb1-b24c-d4c0cfc057f4/dns/0.log"
Nov 03 16:45:48.613109 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:48.612781    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kgdhx_5806796f-e98d-4eb1-b24c-d4c0cfc057f4/kube-rbac-proxy/0.log"
Nov 03 16:45:48.642231 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:48.642203    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7f6nb_13ece33e-8a7b-418f-aa9e-eaa2fed48273/dns-node-resolver/0.log"
Nov 03 16:45:49.358136 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:49.358105    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j5r6g_e3b0f6f7-b58c-49a1-aa66-57cee4ae2998/node-ca/0.log"
Nov 03 16:45:50.948197 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:50.948170    6506 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-tq6lp"
Nov 03 16:45:52.799270 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:52.799239    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864dbfdb86-4kdzq_cbeb1f8a-f73e-4b82-bdc5-048c3e57453e/kube-auth-proxy/0.log"
Nov 03 16:45:52.965496 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:52.965464    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-9z9vz_2cf76626-1d92-4255-8343-5ae7f38a01ab/istio-proxy/1.log"
Nov 03 16:45:52.965719 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:52.965700    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-9z9vz_2cf76626-1d92-4255-8343-5ae7f38a01ab/istio-proxy/0.log"
Nov 03 16:45:53.036093 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:53.036018    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-lmt2w_f9b93c18-e065-4198-897d-2d7dcdcc8d07/istio-proxy/0.log"
Nov 03 16:45:53.081840 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:53.081723    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-pq785_9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f/istio-proxy/0.log"
Nov 03 16:45:53.156681 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:53.156649    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-z5jhm_da47afac-7b95-44ce-867c-1610ca16c13a/istio-proxy/0.log"
Nov 03 16:45:53.157317 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:53.157295    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-z5jhm_da47afac-7b95-44ce-867c-1610ca16c13a/istio-proxy/1.log"
Nov 03 16:45:54.929430 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:54.929395    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8vvpp_a9f7db5a-c0f7-480e-a19f-8b77e7285af2/serve-healthcheck-canary/0.log"
Nov 03 16:45:55.696990 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:55.696961    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hdzxp_61353437-1060-48e2-89f0-b51beb1fdb60/kube-rbac-proxy/0.log"
Nov 03 16:45:55.726948 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:55.726921    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hdzxp_61353437-1060-48e2-89f0-b51beb1fdb60/exporter/0.log"
Nov 03 16:45:55.756361 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:55.756338    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hdzxp_61353437-1060-48e2-89f0-b51beb1fdb60/extractor/0.log"
Nov 03 16:45:56.539421 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:56.539393    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864dbfdb86-4kdzq_cbeb1f8a-f73e-4b82-bdc5-048c3e57453e/kube-auth-proxy/0.log"
Nov 03 16:45:56.625895 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:56.625862    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-9z9vz_2cf76626-1d92-4255-8343-5ae7f38a01ab/istio-proxy/1.log"
Nov 03 16:45:56.626104 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:56.626089    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-9z9vz_2cf76626-1d92-4255-8343-5ae7f38a01ab/istio-proxy/0.log"
Nov 03 16:45:56.659128 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:56.659099    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-lmt2w_f9b93c18-e065-4198-897d-2d7dcdcc8d07/istio-proxy/0.log"
Nov 03 16:45:56.679646 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:56.679626    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-pq785_9fc4ed21-1d34-4dd0-9848-e3ebf48fce9f/istio-proxy/0.log"
Nov 03 16:45:56.716635 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:56.716600    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-z5jhm_da47afac-7b95-44ce-867c-1610ca16c13a/istio-proxy/1.log"
Nov 03 16:45:56.716780 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:56.716614    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-z5jhm_da47afac-7b95-44ce-867c-1610ca16c13a/istio-proxy/0.log"
Nov 03 16:45:58.434222 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:58.434191    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-d59cb86d5-fcmb5_cf23827c-bb93-4a99-a7ee-1686558b8a6d/manager/0.log"
Nov 03 16:45:58.456147 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:58.456123    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-d59cb86d5-fcmb5_cf23827c-bb93-4a99-a7ee-1686558b8a6d/copy-manifests/0.log"
Nov 03 16:45:58.798070 ip-10-0-137-159 kubenswrapper[6506]: E1103 16:45:58.798033    6506 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hdzxp" containerName="extractor"
Nov 03 16:45:59.687882 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:59.687853    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-6fb6bc9cbf-rbvsh_be2806bc-640c-4904-8f88-8ccc2d58c6ae/manager/0.log"
Nov 03 16:45:59.721328 ip-10-0-137-159 kubenswrapper[6506]: I1103 16:45:59.721299    6506 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-scbnj_7849bd02-4d3c-45c7-8415-78200374eba7/jobset-operator/0.log"